DDLambda causes thread leak

[junkfactory]

I am integrating aws lambda java with the datadog extension and dd-lambda-java

Following the instructions in the README file on adding

var dd = new DDLambda(context);
try {
    //make money
} catch (Exception e) {
   //no money
  dd.error(context);
} finally {
  dd.finish();
}

Found these exceptions

unable to create native thread: possibly out of memory or process/resource limits reached: java.lang.OutOfMemoryError
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
	at java.base/java.lang.Thread.start0(Native Method)
	at java.base/java.lang.Thread.start(Unknown Source)
	at com.timgroup.statsd.StatsDSender.startWorkers(StatsDSender.java:71)
	at com.timgroup.statsd.NonBlockingStatsDClient.<init>(NonBlockingStatsDClient.java:348)
	at com.timgroup.statsd.NonBlockingStatsDClient.<init>(NonBlockingStatsDClient.java:370)
	at com.timgroup.statsd.NonBlockingStatsDClientBuilder.build(NonBlockingStatsDClientBuilder.java:175)
	at com.datadoghq.datadog_lambda_java.ExtensionMetricWriter.<init>(MetricWriter.java:50)
	at com.datadoghq.datadog_lambda_java.DDLambda.<init>(DDLambda.java:55)
	at com.datadoghq.datadog_lambda_java.DDLambda.<init>(DDLambda.java:66)

Appears https://github.com/DataDog/datadog-lambda-java/blob/main/src/main/java/com/datadoghq/datadog_lambda_java/DDLambda.java#L55 is always creating worker threads from statsD client.

Confirmed with threaddump as well

"StatsD-Processor-1" daemon prio=5 Id=15 TIMED_WAITING


"StatsD-Sender-1" daemon prio=5 Id=16 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@655523dd


"StatsD-Processor-1" daemon prio=5 Id=18 TIMED_WAITING


"StatsD-Sender-1" daemon prio=5 Id=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f7e336b


"StatsD-Processor-1" daemon prio=5 Id=21 TIMED_WAITING


"StatsD-Sender-1" daemon prio=5 Id=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a2b2322


"StatsD-Processor-1" daemon prio=5 Id=23 TIMED_WAITING


"StatsD-Sender-1" daemon prio=5 Id=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5e1218b4


"StatsD-Processor-1" daemon prio=5 Id=25 TIMED_WAITING


"StatsD-Sender-1" daemon prio=5 Id=26 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12958360


"StatsD-Processor-1" daemon prio=5 Id=27 TIMED_WAITING


"StatsD-Sender-1" daemon prio=5 Id=28 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c6e0f32


"StatsD-Processor-1" daemon prio=5 Id=29 TIMED_WAITING

Initializing MetricWriter could naively be in a static initializer but that's just me.

[Perry-Olsson]

I'm also seeing this error after instrumenting APM.

[Perry-Olsson]

Temporary solution could be calling NonBlockingStatsDClient.close() in DDLambda.finish(). This will kill the infinite sendLoop() running in the thread created by StatsDSender.startWorkers()

[DarcyRaynerDD]

Hey everyone, I'd like to apologize for the delay in us responding to this ticket. We will look through this in the next few days.

[VitoMakarevich]

@DarcyRaynerDD any updates? It's really preventing us from using the library as API becomes unstable.

[VitoMakarevich]

This patchset addresses the leak partially, but I'm not an expert in datadog java, so no sure what else consequences it might introduce. Also, I see that from time to time the threads are still left(2 per invocation), but since it's very rare, it's not causing problems.