io.netty.handler.timeout.ReadTimeoutException not getting retried for S3AsyncClient::getObject

[ZhaoMJ]

Describe the bug

We have noticed that S3AsyncClient does not retry io.netty.handler.timeout.ReadTimeoutException:

[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	Caused by: java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null (SDK Attempt Count: 1)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at com.drw.xd.objectstorage.S3InputStream$S3InputStreamEnumeration.nextElement(S3InputStream.java:114)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		... 8 more
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null (SDK Attempt Count: 1)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 		at java.base/java.lang.Thread.run(Thread.java:840)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	Caused by: io.netty.handler.timeout.ReadTimeoutException
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: Caused by: java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null (SDK Attempt Count: 1)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at com.drw.xd.objectstorage.S3InputStream$S3InputStreamEnumeration.nextElement(S3InputStream.java:114)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	... 10 more
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null (SDK Attempt Count: 1)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: 	at java.base/java.lang.Thread.run(Thread.java:840)
[2025-11-22, 08:17:29 UTC] {pod.py:942} INFO - [base] logs: Caused by: [CIRCULAR REFERENCE: io.netty.handler.timeout.ReadTimeoutException]

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

ReadTimeoutException should be retried.

Current Behavior

ReadTimeoutException is not being retried, but thrown as retryPolicyDisallowedRetryException.

Per #686, ReadTimeoutException should be wrapped as an IOException and retried by RetryStrategy. However, decorateException doesn't appear to be used in the call chain.

Reproduction Steps

Keep firing a lot getObject, until io.netty.handler.timeout.ReadTimeoutException is thrown.
Observe that the error is thrown directly without being retried.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.35.11

JDK version used

openjdk version "17.0.10" 2024-01-16 LTS OpenJDK Runtime Environment Corretto-17.0.10.7.1 (build 17.0.10+7-LTS) OpenJDK 64-Bit Server VM Corretto-17.0.10.7.1 (build 17.0.10+7-LTS, mixed mode, sharing)

Operating System and version

Amazon Linux 2

[bhoradc]

Hello @ZhaoMJ

Thank you for reporting the issue. I validated the SDK behavior and can confirm that it correctly retries ReadTimeoutException under default SDK retry settings. The ReadTimeoutException gets wrapped as IOException through standard code path. And since IOException is included in default retryable exceptions, the SDK does retry in my case.

Can you please provide a minimal reproducible code sample and if possible, the SDK Debug logs for us to investigate the issue at our end?

Regards,
Chaitanya

[ZhaoMJ]

@bhoradc Thanks for the response! I took a closer look at NettyRequestExecutor and realized I overlooked a few things and that ReadTimeoutException is indeed getting wrapped as IOException in the channel future listener. Sorry for my false report!

It is weird that we got SDK Attempt Count: 1 then. Our retry strategy is configured as follows. Do you see any obvious issues that could lead to not retrying as expected?

BackoffStrategy backoffStrategy = BackoffStrategy.exponentialDelay(
                Duration.ofMillis(1000), 
                Duration.ofSeconds(20)); // SDK default
        var strategy = SdkDefaultRetryStrategy.standardRetryStrategyBuilder()
                .maxAttempts(S3_RETRIES)
                .backoffStrategy(backoffStrategy)
                .throttlingBackoffStrategy(backoffStrategy)
                .circuitBreakerEnabled(false)
                .build();
        s3AsyncBuilder.overrideConfiguration(o -> o.retryStrategy(strategy));

Here S3_RETRIES is 7.

Unfortunately, I didn't have debug logging enabled. ReadTimeoutException is rare for us. Do you know how to mock this exception so I can try to reproduce it and collect some SDK debug logs?

[ZhaoMJ]

@bhoradc Apologies for the noise. I think I have figured it out - it turns out the issue is SdkDefaultRetryStrategy.standardRetryStrategyBuilder(), which doesn't configure the default retryable exceptions.
I should use AwsRetryStrategy.standardRetryStrategy().toBuilder() instead.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[ZhaoMJ]

@bhoradc sorry I was wrong again since we got another instance where ReadTimeoutException was not being retried with the similar stack trace, after I switched to use AwsRetryStrategy.standardRetryStrategy().toBuilder().

Here is the stack trace:

Caused by: java.util.concurrent.ExecutionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null (SDK Attempt Count: 1)
        at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
        at com.drw.xd.objectstorage.S3InputStream$S3InputStreamEnumeration.nextElement(S3InputStream.java:114)
        ... 33 common frames omitted
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: null (SDK Attempt Count: 1)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
        at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:95)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:139)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:157)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:117)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:78)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$0(MakeAsyncHttpRequestStage.java:108)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
        at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.completeResponseFuture(MakeAsyncHttpRequestStage.java:255)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$executeHttpRequest$3(MakeAsyncHttpRequestStage.java:167)
        at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934)
        at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911)
        at java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: io.netty.handler.timeout.ReadTimeoutException: null

The S3 request is just a simple GetObjectRequest:

GetObjectRequest.Builder builder = GetObjectRequest.builder()
                .bucket(bucket)
                .key(key);

And we read the response as an input stream:

ResponseBytes<GetObjectResponse> response = future.get();
return response.asInputStream();

Judging from the stack trace, it still feels to me that ReadTimeoutException might not be properly wrapped. I'm trying to figure out an MRE. Please let me know if there is anything helpful I can provide.

For now, I think we will just add an explicit .retryOnExceptionOrCause(ReadTimeoutException.class) in our retry strategy.

[bhoradc]

Hi @ZhaoMJ

No worries, thanks for the additional details.

Caused by: io.netty.handler.timeout.ReadTimeoutException: null

In your case, the ReadTimeoutException doesn't seem to get wrapped by NettyUtils.decorateException(), which typically converts it to a retryable IOException.

SdkDefaultRetryStrategy.standardRetryStrategyBuilder() should work by default for retrying ReadTimeoutException when properly wrapped.

What's the behavior if you force a timeout with .readTimeout(Duration.ofMillis(x)) on the HTTP client builder?

If you can kindly provide a minimal reproducible code sample, that helps to further look into this.

Regards,
Chaitanya