SDK ignores "Content-Length" header of a S3 PutObject request with the body backed by an InputStream #2908
Description
Describe the bug
NOTE: This issue is similar to #460, which was fixed in 2.0.0-preview-11. The only difference is that in #460, the body was created from a String, and in this one, the body is created from an InputStream.
If I create a S3 PutObject request using an InputStream and I incorrectly specify the size of the body LESS than the size of the payload, SDK will actually send the entire payload.
private static final int OBJECT_SIZE = 512; // bytes
public static void main(...) throws ... {
S3Client client = createS3Client();
PutObjectRequest req = PutObjectRequest.builder().bucket(BUCKET_NAME).key(KEY).build();
// Create the payload whose size is 512 bytes.
InputStream is = createInputStreamForBody(OBJECT_SIZE);
// Create the request body with the InputStream, but with a wrong `contentLength` 256.
RequestBody body = RequestBody.fromInputStream(is, OBJECT_SIZE / 2);
client.putObject(req, body);
}
/**
* Returns an InputStream backed by a byte array of `size` filled with random data.
*/
private InputStream createInputStreamForBody(int size) {
Random rnd = new Random();
byte[] bytes = new byte[size];
rnd.nextBytes(bytes);
return new ByteArrayInputStream(bytes);
}The SDK sends the header and the first chunk of the body:
PUT /key1 HTTP/1.1
Host: <omitted>.s3.ap-northeast-1.amazonaws.com
amz-sdk-invocation-id: <omitted>
amz-sdk-retry: 0/0/500
Authorization: AWS4-HMAC-SHA256 Credential=<omitted>/20211210/ap-northeast-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-retry;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=<omitted>
Content-Type: application/octet-stream
Expect: 100-continue
User-Agent: aws-sdk-java/2.10.54 Linux/5.10.81 OpenJDK_64-Bit_Server_VM/17.0.1+12-nixos Java/17.0.1 vendor/N_A io/sync http/Apache
x-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
X-Amz-Date: 20211210T073003Z
x-amz-decoded-content-length: 256
Content-Length: 430
Connection: Keep-Alive
200;chunk-signature=<omitted>
.....(200 is the size of the chunk in hexadecimal. 0x200 = 512 bytes)
And Amazon S3 returns the following error response:
HTTP/1.1 400 Bad Request
x-amz-request-id: <omitted>
x-amz-id-2: <omitted>
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Fri, 10 Dec 2021 07:30:02 GMT
Server: AmazonS3
Connection: close
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>IncompleteBody</Code><Message>The request body terminated unexpectedly</Message>
<RequestId><omitted></RequestId><HostId><omitted></HostId></Error>Expected behavior
I would expect that the SDK to stop reading from the InputStream when it has read the contentLength bytes.
Java API: software.amazon.awssdk.core.sync.RequestBody#fromInputStream()
public static RequestBody fromInputStream(InputStream inputStream, long contentLength)Parameters:
inputStream- Input stream to send to the service. The stream will not be closed by the SDK.contentLength- Content length of data in input stream.
Current behavior
The SDK does not respect the contentLength, and tries to send all bytes read from the InputStream. This causes Amazon S3 to return 400 Bad Request error.
Steps to Reproduce
- Create the request body using
RequestBody.fromInputStream(InputStream inputStream, long contentLength)- But make
contentLengthsmaller than the actual size of the payloadinputStream.
- But make
- Send a S3 PutObject request with the request body.
(I will provide a full Java source code later)
Possible Solution
Update the SDK so that it will stop reading from the InputStream when it has read the contentLength bytes.
Context
For example, I may want to send and store only first portion of a local file to Amazon S3, by setting the contentLength smaller than the actual file size.
AWS Java SDK version used
2.10.54 and 2.17.98
JDK version used
openjdk version "11.0.12" 2021-07-20 64-Bit Server VM
Operating System and version
Linux x86_64 (NixOS 22.05pre Quokka)