Provide a hook for configuring Logback programatically

[celcius112]

At the moment we are considering migrating few applications to spring-native. Everything works fine excepts our Logback configuration. We use a logback.xml, and as stated in spring-attic/spring-native#625 it will not work as intended, nor does it seem to be an easy problem to tackle.

We were already considering moving to the programmatic Logback configuration, using a Configurator, which would probably be easier to use with spring-native and would provide better application startup for regular applications. However, we cannot migrate all of our logback.xml configuration since we cannot import default spring configuration nor use springProperty/springProfile elements.

[sdeleuze]

Status update:

• Currently Spring Boot 3 only supports Logback configuration via application.properties or Yaml equivalent, logback.xml is unsupported.
• After a deeper look, I confirm that build time generation of programmatic Logback configuration from logback.xml is the only identified way to support it on native, there are already discussions and experiments in the JVM/GraalVM communities about that.
• Providing such hook to users would unlock those who need more advanced customization capabilities than what application.properties offers and could allow Spring community to experiment about build time logback.xml to programmatic version in an autonomous manner. To be evaluated against other priorities of course.
• Potential link with Logback 1.4 upgrade from Upgrade to SLF4J 2.0 and Logback 1.4 #12649

[philwebb]

It looks like a ServiceLoader could be used to plug-in a custom ch.qos.logback.classic.spi.Configurator, however, I don't think our logging initialization code would detect that happened and we we'd probably end up applying our configuration. The org.springframework.boot.logging.AbstractLoggingSystem class has a initializeWithConventions which attempts to detect logback.xml and if it's found not apply our defaults.

We do already have a package-private org.springframework.boot.logging.logback.DefaultLogbackConfiguration class which applies our defaults in a programmatic way. I don't really want to make that class public, especially if the GraalVM community is working on a logback.xml -> Configurator processor. Presumably, such a processor would need to deal with <include> tags, at which point our existing XML files could just be included and used directly.

Adding hook points for logging is quite a challenge since they're normally needed before the Spring ApplicationContext is loaded. I think we should see how the community experiments progress before we do too much on our side.

[jack-berg]

I'd like to see this issue solved to accommodate what I think is a similar use case:

OpenTelemetry java has logback and log4j appenders which bridge log data from those logging systems in the OpenTelemetry ecosystem. These appenders need to be configured with an OpenTelemetry instance, an object which contains APIs for bridging log records into OpenTelemetry, and which is programmatically configured by the user. The general workflow is for users to initialize an OpenTelemetry instance very early in the application lifecycle, and then use it when installing various bits of instrumentation. In spring environments, its typical to create an OpenTelemetry bean to participate in dependency injection.

The programmatic configuration of the OpenTelemetry instance is at odds with typical static log configuration. But both log4j and logback allow for programmatic configuration as well, where you can modify a log4j.xml or logback.xml configuration at runtime by adding / editing appenders.

It would nice to offer OpenTelemetry users a helper function public static void installOpenTelemetryAppender(OpenTelemetry), which modifies the log4j / logback configuration at runtime after the OpenTelemetry instance has been configured.

This workflow doesn't quite work in spring boot because programmatic modifications to the log4j / logback configuration are erased when spring boot is initialized (I believe this line is the culprit). Users could do the modification after spring initializes, but they'll miss out on valuable logs detailing spring initialization.

It would be great if spring boot could provide a hook for performing programmatic configuration of log4j / logback early enough that any appenders added are able to receive application startup logs.

[wilkinsona]

Thanks for the use case, @jack-berg. Unfortunately, unless I have misunderstood, I don't think what you have described is possible.

In spring environments, its typical to create an OpenTelemetry bean to participate in dependency injection

It would be great if spring boot could provide a hook for performing programmatic configuration of log4j / logback early enough that any appenders added are able to receive application startup logs.

These goals seem to be contradictory. If we're using an OpenTelemetry bean, it will not be created until some point during application context refresh. This is too late to receive (all) application startup logs.

It would nice to offer OpenTelemetry users a helper function public static void installOpenTelemetryAppender(OpenTelemetry), which modifies the log4j / logback configuration at runtime after the OpenTelemetry instance has been configured.

I don't think this belongs in Spring Boot. From what I understand, there's nothing Spring Boot-specific about wanting to configure Logback or Log4j with an Open Telemetry appender. For example, with Logback, it would be done through the standard ch.qos.logback.classic.LoggerContext API. As such, I think it should be part of OpenTelemetry so that it benefits all of its users not just those who are also using spring Boot.

[jack-berg]

Sorry I think I presented my case in a confusing way. I don't think spring boot should in any way be responsible for installing / configuring the opentelemetry appender. Rather, I was just trying to explain why programmatic configuration is important for the OpenTelemetry appenders, and express support for spring boot adding some sort of generic callback or hook that opentelemetry code can leverage to perform programmatic configuration soon after spring boot log configuration, such that the opentelemetry appenders can receive initialization logs.

[philwebb]

This might overlap with Logback's existing META-INF/services/ch.qos.logback.classic.spi.Configurator support.

[making]

I created samples to automatically configure Logback. I'm not sure if it's the right way to do it, but it works for me in the following two use cases:

• Automatically install Otel Logback Appender　https://github.com/making/otel-logs-autoconfigure/blob/develop/src/main/java/am/ik/spring/opentelemetry/logs/logback/LogbackAppenderInstallListener.java
• Automatically configure EcsEncoder https://github.com/making/logback-ecs-encoder-autoconfigure/blob/develop/src/main/java/am/ik/spring/ecs/logback/EcsEncoderConfigurationListener.java

[Sineaggi]

We have a slightly different use-case. We'd be happy with some extension point being a simplified version of what already exists in DefaultLogbackConfiguration. Creating a spring boot starter that integrates with logback via their rollbar-logback package would require a simple set of configuration values (preferably passed in via application.properties) and wouldn't need the application refresh as mentioned in #25847 (comment).

I'm not sure if something like a logback appender factory could be exposed, or the full logger context. But something that would let us programmatically construct an appender

...
public Appender<ILoggingEvent> getAppender(Context/Environment in) {
    var appender = new RollbarAppender();
    appender.setAccessToken(in.getConfig("logback.rollbar.access-token"));
    return appender;
}

then configure that as we currently configure log levels and structured logging options.

As it is right now, we can't easily configure structured logging and our custom appenders, as spring doesn't apply the logging defaults in the presence of a logback.xml file. Also, extending a Configurator does not let us access the spring Environment, like how the StructuredLogEncoder currently can

Environment environment = (Environment) getContext().getObject(Environment.class.getName());
Assert.state(environment != null, "Unable to find Spring Environment in logger context");

and pre-existing loggers are cleared when spring runs it's defaults.

One potentially brittle way to extend existing support I've found would be to sub-class the LogbackLoggingSystem class, let it do it's automatic setup, then follow-up with custom appender registration. It does look like there are a few non-public helper methods that we'd have to copy-paste/re-implement, but otherwise it works.