Fixed an issue that could cause checksum mismatch errors in S3 uploads. #5836
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This error is sometimes encountered when a customer uses: 1. The AsyncS3Client 2. A ChecksumAlgorithm of SHA1 or SHA256 (instead of the default CRC32) 3. Parallel uploads The root cause was the SDK using thread locals to cache the SHA1 or SHA256 message digest implementations. This meant that if a single event loop thread was processing multiple requests, those two requests would use the same digest implementation to calculate the checksum.
zoewangg
reviewed
Jan 29, 2025
|
|
||
| // Avoid over-caching after large traffic bursts. The maximum chosen here is arbitrary. It's also not strictly | ||
| // enforced, since these statements aren't synchronized. | ||
| if (digestCache.size() <= MAX_CACHED_DIGESTS) { |
Contributor
There was a problem hiding this comment.
size() may be expensive, should we track the size using atomic integer instead?
There was a problem hiding this comment.
How would you coordinate the atomic size and the concurrent deque?
Contributor
Author
There was a problem hiding this comment.
I'll switch to a linked blocking deque. Uses locking, but it's still likely faster than creating new checksums and has a constant-time size() method. We can do benchmarking later to verify this is the fastest method.
Is that reasonable?
| DigestThreadLocal(String algorithmName) { | ||
| this.algorithmName = algorithmName; | ||
| /** | ||
| * Retrieve the message digest bytes. This will close the message digest when invoked. This is because the underlying |
Contributor
There was a problem hiding this comment.
Question: where do we close messageDigest in this method?
Contributor
Author
There was a problem hiding this comment.
Good catch, that needs a test added.
jeking3
reviewed
Jan 29, 2025
|
|
||
| // Avoid over-caching after large traffic bursts. The maximum chosen here is arbitrary. It's also not strictly | ||
| // enforced, since these statements aren't synchronized. | ||
| if (digestCache.size() <= MAX_CACHED_DIGESTS) { |
There was a problem hiding this comment.
How would you coordinate the atomic size and the concurrent deque?
| throw new RuntimeException("Unable to fetch message digest instance for Algorithm " | ||
| + algorithmName + ": " + e.getMessage(), e); | ||
| return new CloseableMessageDigest((MessageDigest) digest.get().clone()); | ||
| } catch (CloneNotSupportedException e) { // should never occur |
There was a problem hiding this comment.
Any time a comment says "should never occur", it seems to happen. Why can't this method declare that it throws CloneNotSupportedException instead?
Contributor
Author
There was a problem hiding this comment.
CloneNotSupportedException is a checked exception. Because the SDKs don't throw checked exceptions, the callers would need to wrap it in an unchecked exception themselves.
Contributor
Author
There was a problem hiding this comment.
I've improved the exception message if this scenario does happen.
dagnir
reviewed
Jan 29, 2025
Comment on lines
106
to
107
| // Avoid over-caching after large traffic bursts. The maximum chosen here is arbitrary. It's also not strictly | ||
| // enforced, since these statements aren't synchronized. |
Contributor
There was a problem hiding this comment.
Would it be possible to have a "prefilled" cache of lazy loaded digests, and then when we can't use a cached digest, we have a different instance that just closes and doesn't need to interact with the cache?
Contributor
Author
There was a problem hiding this comment.
That's an option. It might be trickier to decide on the size of the cache, and then it means that not releasing a message digest to the cache will have long-term performance implications if it's one of those special "cached" digests. The advantage to this implementation is that the odd error that could fail to release the digest back to the cache doesn't really hurt.
…tAlgorithm cache.
dagnir
approved these changes
Jan 29, 2025
dagnir
reviewed
Jan 29, 2025
| digestCache.addFirst(digest.get()); | ||
| } | ||
| // Drop this digest is the cache is full. | ||
| digestCache.offerFirst(digest.get()); |
zoewangg
approved these changes
Jan 29, 2025
… sends the payload, and then a retry happens with checksums enabled.
|
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
ghetelgb
pushed a commit
to ghetelgb/aws-sdk-java-v2
that referenced
this pull request
Feb 7, 2025
…s. (aws#5836) * Fixed an issue that could cause checksum mismatch errors in S3 uploads. This error is sometimes encountered when a customer uses: 1. The AsyncS3Client 2. A ChecksumAlgorithm of SHA1 or SHA256 (instead of the default CRC32) 3. Parallel uploads The root cause was the SDK using thread locals to cache the SHA1 or SHA256 message digest implementations. This meant that if a single event loop thread was processing multiple requests, those two requests would use the same digest implementation to calculate the checksum. * Fixed an issue where SignerUtils was using the internal class of another module. * Update clone() failure message. Move to LinkedBlockingDeque for DigestAlgorithm cache. * Added direct testing of the DigestAlgorithm cache. * Fixed a transient issue that could occur when the first attempt fully sends the payload, and then a retry happens with checksums enabled.
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Jul 23, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
6 tasks
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Jul 29, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Aug 11, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Aug 11, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Aug 18, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Aug 18, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
imsayari404
added a commit
to imsayari404/presto
that referenced
this pull request
Aug 20, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
hantangwangd
pushed a commit
to prestodb/presto
that referenced
this pull request
Aug 20, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 2, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 8, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 8, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 8, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 8, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 8, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
lga-zurich
pushed a commit
to lga-zurich/presto-exchange
that referenced
this pull request
Sep 8, 2025
Disable checksum validation and chunked encoding to ensure compatibility with MinIO and newer AWS SDK versions per aws/aws-sdk-java-v2#5836
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This error is sometimes encountered when a customer uses:
The root cause was the SDK using thread locals to cache the SHA1 or SHA256 message digest implementations. This meant that if a single event loop thread was processing multiple requests, those two requests would use the same digest implementation to calculate the checksum.
This PR updates the SHA1 and SHA256 (and MD5, though it's not used by S3) to use a LIFO cache.