DynamoDB connection pool tied up when interrupting #2936
Description
Describe the bug
In my service, I was time limiting a block of code which involved dynamodb queries and eventually after enough timeouts, I am seeing the following error: SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool. It appears that if i interrupt the ddb client, then the connection is forever tied up and there is no available connection to make further calls
After several runs, i noticed that if i run future.cancel(false) (no interrupt) vs future.cancel(true) the service remains stable, but the threads are allowed to finish. I verified with the java sdk client metrics that LeasedConcurrency goes up and never goes back down.
Expected behavior
When ddb client aborts due to interrupt (AbortedException), the http connection is released.
Current behavior
When ddb client aborts due to interrupt, the http connection is NOT released therefore eventually depleting the http pool connection.
Steps to Reproduce
Something like this:
final Future future = executorService.submit(() -> {
// some longer running ddb calls
}
final Object response;
try {
response = future.get(MS_LIMIT_TO_RESPOND, TimeUnit.MILLISECONDS);
} catch (TimeoutException exception) {
LOG.warn("Took longer than allotted {} ms to generate response.", MS_LIMIT_TO_RESPOND, exception);
} catch (InterruptedException | ExecutionException exception) {
LOG.error("Failed to generate response", exception);
} finally {
// If .cancel(true), then the thread will try to be interrupted, causing the issue.
future.cancel(true);
LOG.info("Returning response {}", response);
}
Possible Solution
Other tickets i saw with connection pool timeouts were regarding s3 and closing the object to ensure connection is released. I think upon sdk AbortedException or whatever exception for interrupt, the ddb connection should be closed.
Context
I was trying to limit the execution time on my service. If it didn't complete within a time limit, it would return an empty response.
AWS Java SDK version used
2
JDK version used
1.8
Operating System and version
Amazon Linux