Skip to content

Deprecate old HTTP wrapping methods and add instructions for installing dd-trace-java. #31

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Feb 24, 2021
Merged

Deprecate old HTTP wrapping methods and add instructions for installing dd-trace-java. #31

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
79244b7
Deprecate old HTTP wrapping methods and start on trace/log correlatio…
Feb 19, 2021
1c3cf98
Check for traceId 0
Feb 19, 2021
ee02673
Change distributed tracing notes
Feb 19, 2021
a08342e
Add trace/log correlation notes
Feb 19, 2021
e564f01
README.md
Feb 19, 2021
eaf1fe6
Publish latest tracer layer
Feb 23, 2021
4ae7735
Merge branch 'main' into chris.agocs/revise_docs_for_tracer_release
agocs Feb 24, 2021
a46e4cd
Add more docs, bump version
Feb 24, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
227 changes: 84 additions & 143 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ to the Datadog API.

## Installation

This library will be distributed through JFrog [Bintray](https://bintray.com/beta/#/datadog/datadog-maven/datadog-lambda-java). Follow the [installation instructions](https://docs.datadoghq.com/serverless/installation/java/), and view your function's enhanced metrics, traces and logs in Datadog.
This library will be distributed through JFrog [Bintray](https://bintray.com/beta/#/datadog/datadog-maven/datadog-lambda-java).
Follow the [installation instructions](https://docs.datadoghq.com/serverless/installation/java/), and view your function's enhanced metrics, traces and logs in Datadog.

## Environment Variables

Expand All @@ -38,191 +39,131 @@ Once [installed](#installation), you should be able to submit custom metrics fro

Check out the instructions for [submitting custom metrics from AWS Lambda functions](https://docs.datadoghq.com/integrations/amazon_lambda/?tab=java#custom-metrics).

## Distributed Tracing
## Installing the Java Tracer

Wrap your outbound HTTP requests with trace headers to see your lambda in context in APM.
The Lambda Java Client Library provides instrumented HTTP connection objects as well as helper methods for
instrumenting HTTP connections made with any of the following libraries:
The [Java Tracer](https://docs.datadoghq.com/tracing/setup_overview/setup/java/?tab=containers)
is an optional component that allows you to trace the execution of your Java Lambda function.
The traces will be viewable from your Serverless function details page within Datadog.

- java.net.HttpUrlConnection
- Apache HTTP Client
- OKHttp3
Briefly, in order to use the Java tracer, the following prerquisites must be met (detailed below):
1. The `datadog-lambda-java` library must be included in your project, following these [installation instructions](https://docs.datadoghq.com/serverless/installation/java/)
1. You must use a compatible Java runtime
1. You must attach the Java Tracer lambda layer
1. Several environment variables must be set
1. Your handler must instantiate a `new DDLambda` in order to start traces, and call `DDLambda#finish` in order to end traces.


Don't see your favorite client? Open an issue and request it. Datadog is adding to
this library all the time.
### Cold start considerations

### HttpUrlConnection examples
The Java Tracer adds a nontrivial cold start penalty.
Expect roughly 6 seconds per cold start if your Lambda function is configured with 3008MB of memory.
Lambda runtime CPU scales with the amount of memory allocated, so allocating more memory may help alleviate cold start issues.
Also consider using provisioned concurrency to keep your lambda function warm.

```java
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
DDLambda dd = new DDLambda(request, lambda);

URL url = new URL("https://example.com");
HttpURLConnection instrumentedUrlConnection = dd.makeUrlConnection(url); //Trace headers included
### Compatible Java runtimes

instrumentedUrlConnection.connect();

return 7;
}
}
```

Alternatively, if you want to do something more complex:
- java8.al2 (aka Java 8 (Corretto))
- java11 (aka Java 11 (Corretto))

```java
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
DDLambda dd = new DDLambda(request, lambda);

URL url = new URL("https://example.com");
HttpURLConnection hc = (HttpURLConnection)url.openConnection();
If your lambda function is using Java 8, please change it to Java 8 Corretto.
It's called java8.al2 if you're editing serverless.yaml.

//Add the distributed tracing headers
hc = (HttpURLConnection) dd.addTraceHeaders(hc);
### Required Lambda Layer containing the Java Tracer

hc.connect();

return 7;
}
}
```
arn:aws:lambda:[REGION]:464622532012:layer:dd-trace-java:2
```

### Apache HTTP Client examples
The lambda layer version (in this case, `2`) will always correspond with the minor version of the `datadog-lambda-java` library.

```java
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
DDLambda dd = new DDLambda(request, lambda);

HttpClient client = HttpClientBuilder.create().build();

HttpGet hg = dd.makeHttpGet("https://example.com"); //Trace headers included

HttpResponse hr = client.execute(hg);
return 7;
}
}
### Required environment variables for the Java Tracer

```bash
JAVA_TOOL_OPTIONS: "-javaagent:\"/opt/java/lib/dd-java-agent.jar\""
DD_LOGS_INJECTION: "true"
DD_JMXFETCH_ENABLED: "false"
DD_TRACE_ENABLED: "true"
```

Alternatively, if you want to do something more complex:
### Required code modification for the Java Tracer

In order to use the Java Tracer, you must instantiate a new `DDLambda` at the beginning of your Lambda function and call `DDLambda#finish()` at the end of it.

```java
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
DDLambda dd = new DDLambda(request, lambda);

HttpClient client = HttpClientBuilder.create().build();
HttpGet hg = new HttpGet("https://example.com");

//Add the distributed tracing headers
hg = (HttpGet) dd.addTraceHeaders(hg);

HttpResponse hr = client.execute(hg);
return 7;
}
}
```
public class Handler{

public ApiGatewayResponse handleRequest(APIGatewayProxyRequestEvent input, Context context){
DDLambda ddl = new DDLambda(input, context); // required to set various tags inside the tracer

### OKHttp3 Client examples
ddl.metric("foo.bar", 42, null);
do_some_stuff();
make_some_http_requests();

```java
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
DDLambda dd = new DDLambda(request, lambda);

HttpClient client = HttpClientBuilder.create().build();
OkHttpClient okHttpClient = new OkHttpClient.Builder().build();
Request okHttpRequest = dd.makeRequestBuilder() // Trace headers included
.url("https://example.com")
.build();

Response resp = okHttpClient.newCall(okHttpRequest).execute();

return 7;
}
ddl.finish(); // Required to complete the trace
return new ApiGatewayResponse();
}
}
```

Alternatively:
`DDLambda ddl = new DDLambda(input, context);` starts a new trace (if the Java Tracer agent is attached to the JRE)
and sets tags based on the Lambda context. If there is a trace context attached to the request, that will be used
to set the trace ID and the parent of the span.

```java
public class Handler implements RequestHandler<APIGatewayV2ProxyRequestEvent, APIGatewayV2ProxyResponseEvent> {
public Integer handleRequest(APIGatewayV2ProxyRequestEvent request, Context context){
DDLambda dd = new DDLambda(request, lambda);

HttpClient client = HttpClientBuilder.create().build();
OkHttpClient okHttpClient = new OkHttpClient.Builder().build();
Request okHttpRequest = new Request.Builder()
.url("https://example.com")
.build();

//Add the distributed tracing headers
okHttpRequest = dd.addTraceHeaders(okHttpRequest);

Response resp = okHttpClient.newCall(okHttpRequest).execute();

return 7;
}
}
```
`ddl.finish();` finishes the active span and closes the active trace scope.
The tracer will flush the trace to Cloudwatch logs once this is called.

### Trace/Log Correlations
# Distributed Tracing

In order to correlate your traces with your logs, you must inject the trace context
into your log messages. We've added the these into the slf4j MDC under the key `dd.trace_context`
and provided convenience methods to get it automatically. The trace context is added to the MDC as a side
effect of instantiating any `new DDLambda(...)`.
## Upstream Requests

This is an example trace context: `[dd.trace_id=3371139772690049666 dd.span_id=13006875827893840236]`
You may want to include this Lambda invocation as a child span of some larger trace.
If so, you should anticipate that the event triggering the Lambda will have some trace context attached to it.
If this is the case, then you MUST instantiate `DDLambda` with both the request and the lambda context in order for it to extract the trace context.
E.g. `DDLambda ddl = new DDLambda(request, context);`.
Currently supported events are:

#### JSON Logs
- `APIGatewayProxyRequestEvent`
- `APIGatewayV2ProxyRequestEvent`

If you are using JSON logs, add the trace ID and span ID to each log message with the keys
`dd.trace_id` and `dd.span_id` respectively. To get a map containing trace and span IDs,
call `DDLambda.getTraceContext()`. Union this map with the JSON data being logged.
If you are using a different event with trace context, you may choose to create a class that implements `Headerable` and supply that as the event instead.

#### Plain text logs
## Downstream Requests

If you are using plain text logs, then you must create a new [Parser](https://docs.datadoghq.com/logs/processing/parsing/?tab=matcher)
by cloning the existing Lambda Pipeline. The new parser can extract the trace context from the correct position in the logs.
Use the helper `_trace_context` to extract the trace context. For example, if your log line looked like:
The dd-trace-java tracer will automatically add trace context to outgoing requests for a number of popular services.
The list of instrumented services can be found here: https://docs.datadoghq.com/tracing/setup_overview/compatibility_requirements/java/ .
If you wish to enable a beta integration, please note that you must do so using an environment variable.

```
INFO 2020-11-11T14:00:00Z LAMBDA_REQUEST_ID [dd.trace_id=12345 dd.span_id=67890] This is a log message
```
# Trace/Log Correlation

Then your parser rule would look like:
Please see [Connecting Java Logs and Traces](https://docs.datadoghq.com/tracing/connect_logs_and_traces/java/?tab=log4j2)

```
my_custom_rule \[%{word:level}\]?\s+%{_timestamp}\s+%{notSpace:lambda.request_id}%{_trace_context}?.*
```
In brief, if you set the environment variable `DD_LOGS_INJECTION=true`, your trace ID and span ID are automatically injected into the MDC.
If you are using JSON-formatted logs and logging using Logback, there is nothing left to do.

#### Log4j / SLF4J
## Raw-formatted logs (Log4J, etc.)

We have added the Trace ID into the slf4j MDC under the key `dd.trace_context`. That can be accessed
using the `%X{dd.trace_context}` operator. Here is an example `log4j.properties`:
For raw formatted logs, you must update your log format and your log parser. Instructions for both are below.

```
log = .
log4j.rootLogger = DEBUG, LAMBDA
### Log Format

log4j.appender.LAMBDA=com.amazonaws.services.lambda.runtime.log4j.LambdaAppender
log4j.appender.LAMBDA.layout=org.apache.log4j.PatternLayout
log4j.appender.LAMBDA.layout.conversionPattern=%d{yyyy-MM-dd HH:mm:ss} %X{dd.trace_context} %-5p %c:%L - %m%n
If you are using raw formatted logs, update your formatter to include `dd.trace_context`. E.g.

```xml
<Pattern>"%d{yyyy-MM-dd HH:mm:ss} <%X{AWSRequestId}> %-5p %c:%L %X{dd.trace_context} %m%n"</Pattern>
<!--Please note: Request ID Trace Context -->
```

would result in log lines looking like `2020-11-13 19:21:53 [dd.trace_id=1168910694192328743 dd.span_id=3204959397041471598] INFO com.serverless.Handler:20 - Test Log Message`
Please note that `RequestId` has also been added.
RequestId is not strictly necessary for Trace/Log correlation, but it is useful for correlating logs and invocations.

Just like the **Plain Text Logs** in the previous section, you must create a new [Parser](https://docs.datadoghq.com/logs/processing/parsing/?tab=matcher) in order to parse the trace context correctly.

### Grok Parser

#### Other logging solutions
The following grok parser parses Java logs formatted using the pattern in the previous section.

If you are using a different logging solution, the trace ID can be accessed using the method
`DDLambda.getTraceContextString()`. That returns your trace ID as a string that can be added
to any log message.
```
java_tracer %{date("yyyy-MM-dd HH:mm:ss"):timestamp}\s\<%{uuid:lambda.request_id}\>\s%{word:level}\s+%{data:call_site}%{_trace_context}%{data:message}
```
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this parser the same one we provide out of the box with the default Lambda log pipeline? If that is the case, shouldn't this just work without the customer having to know about the parser? If that is not the case, is there a reason we can't format the logs in the format already understood by the default parser?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't have the power (as far as I know) to change how the customer formats their logs. We can just suggest formats. I think the smart thing to do is to bias towards action, get this out there, and see how customers use it.


## Opening Issues

Expand Down
2 changes: 1 addition & 1 deletion build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ sourceCompatibility = 1.8
targetCompatibility = 1.8

group = 'com.datadoghq'
version= '0.0.11'
version= '0.2.0'

allprojects {
repositories {
Expand Down
Loading