Skip to content

feat: Json layout modern implementation #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 27, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 99 additions & 3 deletions docs/core/logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,23 @@ Logging provides an opinionated logger with output structured as JSON.

## Initialization

Powertools extends the functionality of Log4J. Below is an example `#!xml log4j2.xml` file, with the `#!java LambdaJsonLayout` configured.
Powertools extends the functionality of Log4J. Below is an example `#!xml log4j2.xml` file, with the `JsonTemplateLayout` using `#!json LambdaJsonLayout.json` configured.

!!! info "LambdaJsonLayout is now deprecated"

Configuring utiltiy using `<LambdaJsonLayout/>` plugin is deprecated now. While utility still supports the old configuration, we strongly recommend upgrading the
`log4j2.xml` configuration to `JsonTemplateLayout` instead. [JsonTemplateLayout](https://logging.apache.org/log4j/2.x/manual/json-template-layout.html) is recommended way of doing structured logging.

Please follow [this guide](#upgrade-to-jsontemplatelayout-from-deprecated-lambdajsonlayout-configuration-in-log4j2xml) for upgrade steps.

=== "log4j2.xml"

```xml hl_lines="5"
<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="com.amazonaws.services.lambda.runtime.log4j2">
<Configuration>
<Appenders>
<Console name="JsonAppender" target="SYSTEM_OUT">
<LambdaJsonLayout compact="true" eventEol="true"/>
<JsonTemplateLayout eventTemplateUri="classpath:LambdaJsonLayout.json" />
</Console>
</Appenders>
<Loggers>
Expand Down Expand Up @@ -123,6 +130,44 @@ to customise what is logged.
}
```

### Customising fields in logs

- Utility by default emits `timestamp` field in the logs in format `yyyy-MM-dd'T'HH:mm:ss.SSSZz` and in system default timezone.
If you need to customize format and timezone, you can do so by configuring `log4j2.component.properties` and configuring properties as shown in example below:

=== "log4j2.component.properties"

```properties hl_lines="1 2"
log4j.layout.jsonTemplate.timestampFormatPattern=yyyy-MM-dd'T'HH:mm:ss.SSSZz
log4j.layout.jsonTemplate.timeZone=Europe/Oslo
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to actually have that default to UTC unless overridden?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rite now the behaviour is consistent with existing implementation, meaning it will pick system default timezone. But with new approach, users have ability to customise timezone if need be, which was not possible before.

```

- Utility also provides sample template for [Elastic Common Schema(ECS)](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) layout.
The field emitted in logs will follow specs from [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) together with field captured by utility as mentioned [above](#standard-structured-keys).

Use `LambdaEcsLayout.json` as `eventTemplateUri` when configuring `JsonTemplateLayout`.

=== "log4j2.xml"

```xml hl_lines="5"
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="JsonAppender" target="SYSTEM_OUT">
<JsonTemplateLayout eventTemplateUri="classpath:LambdaEcsLayout.json" />
</Console>
</Appenders>
<Loggers>
<Logger name="JsonLogger" level="INFO" additivity="false">
<AppenderRef ref="JsonAppender"/>
</Logger>
<Root level="info">
<AppenderRef ref="JsonAppender"/>
</Root>
</Loggers>
</Configuration>
```

## Setting a Correlation ID

You can set a Correlation ID using `correlationIdPath` attribute by passing a [JSON Pointer expression](https://datatracker.ietf.org/doc/html/draft-ietf-appsawg-json-pointer-03){target="_blank"}.
Expand Down Expand Up @@ -417,3 +462,54 @@ via `samplingRate` attribute on annotation.
Variables:
POWERTOOLS_LOGGER_SAMPLE_RATE: 0.5
```


## Upgrade to JsonTemplateLayout from deprecated LambdaJsonLayout configuration in log4j2.xml

Prior to version [1.10.0](https://github.com/awslabs/aws-lambda-powertools-java/releases/tag/v1.10.0), only supported way of configuring `log4j2.xml` was via `<LambdaJsonLayout/>`. This plugin is
deprecated now and will be removed in future version. Switching to `JsonTemplateLayout` is straight forward.

Below examples shows deprecated and new configuration of `log4j2.xml`.

=== "Deprecated configuration of log4j2.xml"

```xml hl_lines="5"
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="JsonAppender" target="SYSTEM_OUT">
<LambdaJsonLayout compact="true" eventEol="true"/>
</Console>
</Appenders>
<Loggers>
<Logger name="JsonLogger" level="INFO" additivity="false">
<AppenderRef ref="JsonAppender"/>
</Logger>
<Root level="info">
<AppenderRef ref="JsonAppender"/>
</Root>
</Loggers>
</Configuration>
```

=== "New configuration of log4j2.xml"

```xml hl_lines="5"
<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
<Appenders>
<Console name="JsonAppender" target="SYSTEM_OUT">
<JsonTemplateLayout eventTemplateUri="classpath:LambdaJsonLayout.json" />
</Console>
</Appenders>
<Loggers>
<Logger name="JsonLogger" level="INFO" additivity="false">
<AppenderRef ref="JsonAppender"/>
</Logger>
<Root level="info">
<AppenderRef ref="JsonAppender"/>
</Root>
</Loggers>
</Configuration>
```

5 changes: 5 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,11 @@
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-layout-template-json</artifactId>
<version>${log4j.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-jcl</artifactId>
Expand Down
5 changes: 4 additions & 1 deletion powertools-logging/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,10 @@
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>

<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-layout-template-json</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
import com.fasterxml.jackson.databind.JsonMappingException;
import com.fasterxml.jackson.databind.ObjectWriter;

@Deprecated
abstract class AbstractJacksonLayoutCopy extends AbstractStringLayout {

protected static final String DEFAULT_EOL = "\r\n";
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@
import java.util.HashSet;
import java.util.Set;

@Deprecated
abstract class JacksonFactoryCopy {

static class JSON extends JacksonFactoryCopy {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@
import static java.time.Instant.ofEpochMilli;
import static java.time.format.DateTimeFormatter.ISO_ZONED_DATE_TIME;

/***
* Note: The LambdaJsonLayout should be considered to be deprecated. Please use JsonTemplateLayout instead.
*/
@Deprecated
@Plugin(name = "LambdaJsonLayout", category = Node.CATEGORY, elementType = Layout.ELEMENT_TYPE, printObject = true)
public final class LambdaJsonLayout extends AbstractJacksonLayoutCopy {
private static final String DEFAULT_FOOTER = "]";
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
package software.amazon.lambda.powertools.logging.internal;

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolver;
import org.apache.logging.log4j.layout.template.json.util.JsonWriter;
import org.apache.logging.log4j.util.ReadOnlyStringMap;

final class PowertoolsResolver implements EventResolver {

private final EventResolver internalResolver;

PowertoolsResolver() {
internalResolver = new EventResolver() {
@Override
public boolean isResolvable(LogEvent value) {
ReadOnlyStringMap contextData = value.getContextData();
return null != contextData && !contextData.isEmpty();
}

@Override
public void resolve(LogEvent logEvent, JsonWriter jsonWriter) {
StringBuilder stringBuilder = jsonWriter.getStringBuilder();
// remove dummy field to kick inn powertools resolver
stringBuilder.setLength(stringBuilder.length() - 4);

// Inject all the context information.
ReadOnlyStringMap contextData = logEvent.getContextData();
contextData.forEach((key, value) -> {
jsonWriter.writeSeparator();
jsonWriter.writeString(key);
stringBuilder.append(':');
jsonWriter.writeValue(value);
});
}
};
}

static String getName() {
return "powertools";
}

@Override
public void resolve(LogEvent value, JsonWriter jsonWriter) {
internalResolver.resolve(value, jsonWriter);
}

@Override
public boolean isResolvable(LogEvent value) {
return internalResolver.isResolvable(value);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
package software.amazon.lambda.powertools.logging.internal;

import org.apache.logging.log4j.core.LogEvent;
import org.apache.logging.log4j.core.config.plugins.Plugin;
import org.apache.logging.log4j.core.config.plugins.PluginFactory;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverContext;
import org.apache.logging.log4j.layout.template.json.resolver.EventResolverFactory;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolver;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverConfig;
import org.apache.logging.log4j.layout.template.json.resolver.TemplateResolverFactory;

@Plugin(name = "PowertoolsResolverFactory", category = TemplateResolverFactory.CATEGORY)
public final class PowertoolsResolverFactory implements EventResolverFactory {

private static final PowertoolsResolverFactory INSTANCE = new PowertoolsResolverFactory();

private PowertoolsResolverFactory() {}

@PluginFactory
public static PowertoolsResolverFactory getInstance() {
return INSTANCE;
}

@Override
public String getName() {
return PowertoolsResolver.getName();
}

@Override
public TemplateResolver<LogEvent> create(EventResolverContext context,
TemplateResolverConfig config) {
return new PowertoolsResolver();
}
}
52 changes: 52 additions & 0 deletions powertools-logging/src/main/resources/LambdaEcsLayout.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
{
"@timestamp": {
"$resolver": "timestamp",
"pattern": {
"format": "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'",
"timeZone": "UTC"
}
},
"ecs.version": "1.2.0",
"log.level": {
"$resolver": "level",
"field": "name"
},
"message": {
"$resolver": "message",
"stringified": true
},
"process.thread.name": {
"$resolver": "thread",
"field": "name"
},
"log.logger": {
"$resolver": "logger",
"field": "name"
},
"labels": {
"$resolver": "mdc",
"flatten": true,
"stringified": true
},
"tags": {
"$resolver": "ndc"
},
"error.type": {
"$resolver": "exception",
"field": "className"
},
"error.message": {
"$resolver": "exception",
"field": "message"
},
"error.stack_trace": {
"$resolver": "exception",
"field": "stackTrace",
"stackTrace": {
"stringified": true
}
},
"": {
"$resolver": "powertools"
}
}
Loading