S3TransferManager with CRT is failing with throttling issue #4048
Description
Describe the bug
our storage structure in S3 - YYYY/MM/DD/FOLDER1/SUBFOLDER1/EVENTID/EVENT_REQ and YYYY/MM/DD/FOLDER1/SUBFOLDER1/EVENTID/EVENT_RES. For every event requests and responses are loaded into S3 before event gets published to broker.
YYYY/MM/DD/FOLDER1/SUBFOLDER1/EVENTID/ is the prefix and EVENT_REQ and EVENT_RES are object names.
Expected Behavior
We would like to have successful upload to S3 at load.
Current Behavior
We see lines - software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: Response code indicates throttling
which seems to be coming from S3CrtResponseHandlerAdapter
private void handleError(int crtCode, int responseStatus, byte[] errorPayload) {
if (isErrorResponse(responseStatus) && errorPayload != null) {
onErrorResponseComplete(errorPayload);
} else {
SdkClientException sdkClientException =
SdkClientException.create("Failed to send the request: " +
CRT.awsErrorString(crtCode));
failResponseHandlerAndFuture(sdkClientException);
}
}
Reproduction Steps
Our S3TransferManager config code below, with this setting when we do around 3500 PUT/GET request per second to S3 we see the issue in our spring boot project
public S3TransferManager s3TransferManager(AwsCredentialsProvider awsCredentialsProvider) {
S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder()
.credentialsProvider(awsCredentialsProvider)
.maxConcurrency(200)
.checksumValidationEnabled(true)
.targetThroughputInGbps(10.0)
.minimumPartSizeInBytes(1500 * KB)
.build();
S3TransferManager s3TransferManager = S3TransferManager.builder()
.s3Client(s3AsyncClient)
.executor(executor())
.build();
}
private Executor executor() {
ThreadPoolExecutor executor = new ThreadPoolExecutor(0, 300,
60, TimeUnit.SECONDS,
new LinkedBlockingQueue<>(1200),
new ThreadFactoryBuilder().threadNamePrefix("exec").build());
executor.allowCoreThreadTimeOut(true);
return executor;
}
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.0
JDK version used
18
Operating System and version
Linux