Cross region S3 file read is not working without a write or list request

[yuvanakannan]

Describe the bug

We are using AWS SDK V2 and CRT to read, list and write files to S3 buckets across regions (cross region enabled)
We are able to download / read the file from the primary region without any issues.
But when we tried to download from the S3 bucket in other region, it fails with error like
"s.a.a.s.s.m.S3Exception: null (Service: S3, Status Code: 400, Request ID: *****, Extended Request ID: ****)
s.a.a.s.s.m.S3Exception$BuilderImpl.build(S3Exception.java:104)"

But if we issue a list or a write operation, it does not error out. Interestingly if we execute the read/get after a list or write, read /get operation works fine.

It fails if we are directly reading (get object) file from cross region S3 bucket with out executing a list or write operation.

Expected Behavior

Download of the file from a cross region S3 bucket should work fine.

Current Behavior

It fails if we are directly reading (get object) file from cross region S3 bucket with out executing a list or write operation.

Reproduction Steps

// Downloading file
S3TransferManager.builder().s3Client(S3AsyncClient.crtBuilder().region(Region.of("us-east-1")).crossRegionAccessEnabled(true) .build()).build().downloadFile(DownloadFileRequest.builder().getObjectRequest(b -> b.bucket(s3BucketInUsw2).key(s3Key)).destination(file).build()).completionFuture().join();

Possible Solution

As a work around, we are issuing the list operation before reading the file. Also disabling the CRT works.

Additional Information/Context

aws-crt version 0.27.7. Also tested with latest AWS SDK versions - 2.21.28 and CRT 0.28.8

AWS Java SDK version used

2.21.4

JDK version used

17

Operating System and version

linux

[debora-ito]

@yuvanakannan thank for for reaching out, I can repro the error on my side. We'll investigate.

In the meantime, you can use the workarounds you described. I also found that if you remove the region from the S3AsyncClient.crtBuilder(), the downloadFile works fine.

[patricia-valdez]

@yuvanakannan thank for for reaching out, I can repro the error on my side. We'll investigate.

In the meantime, you can use the workarounds you described. I also found that if you remove the region from the S3AsyncClient.crtBuilder(), the downloadFile works fine.

I tried removing the region when building the CRT S3 Async Client as suggested although I got below exception. If I set the region anywhere else, I end up with the same issue with download file

software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@6c8b6782: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@1435a8ca: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@7c590039: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@204b1903: Unable to contact EC2 metadata service.]

[debora-ito]

@patricia-valdez I apologize, when I tested without setting a region in the client, I forgot that I have a default region set in my .aws/config file. My default region is us-west-2, so maybe the error is caused when the us-east-1 is used in the client.

[joviegas]

This issue is happening because CRT internally does a HedObject request and for cross region calls for Clients in us-east-1 region the s3 returns

http-outgoing-0 << "HTTP/1.1 400 Bad Request[\r][\n]"
http-outgoing-0 << "x-amz-request-id: 12wesxx[\r][\n]"
http-outgoing-0 << "x-amz-id-2: someId/wnIvK8KI/Tkf0=[\r][\n]"
http-outgoing-0 << "Content-Type: application/xml[\r][\n]"
http-outgoing-0 << "Date: Wed, 20 Dec 2023 21:41:55 GMT[\r][\n]"
http-outgoing-0 << "Server: AmazonS3[\r][\n]"
http-outgoing-0 << "Connection: close[\r][\n]"
http-outgoing-0 << "[\r][\n]"

In this case S3 is not sending cross region info in x-amz-bucket-region of error response because of which SDK does not know the target region.
I am checking with S3 team to understand why is this issue seen only for "us-east-1" and "HeadObjectApi".

Tag P110014307