Add LegacyMd5Plugin for MD5 checksum calculations in S3 operations requiring checksums #6055

joviegas · 2025-04-24T21:39:20Z

Motivation and Context

The recent S3 changes introduced challenges for customers using S3A (Apache Spark, Iceberg) with various third-party storage providers.
While some providers like MinIO have fixes available, many customers use different storage solutions ( or cannot easily upgrade their third-party stores.
This plugin provides a backwards-compatible solution allowing customers to maintain MD5 checksum calculations where required while adopting S3 Motorcade features.
Note: This Plugin adds MD5 checksum for operations which require checksums along with the default checksum provided by SDK for the operation which require checksum .

 // For synchronous S3 client
 S3Client s3Client = S3Client.builder()
                            .addPlugin(LegacyMd5Plugin.create())
                            .build();

 // For asynchronous S3 client
 S3AsyncClient asyncClient = S3AsyncClient.builder()
                                         .addPlugin(LegacyMd5Plugin.create())
                                         .build();

If you want to add MD5 checksums to the operations that require checksums and want to skip adding of SDK Default checksums for operations that support checksums but not required, then you can enable ClientBuilder options requestChecksumCalculation and responseChecksumValidation as WHEN_REQUIRED, this will add SDK default checksums only to operation that required checksums

// Use LegacyMd5Plugin with requestChecksumCalculation and responseChecksumValidation set to WHEN_REQUIRED
S3AsyncClient asyncClient = S3AsyncClient.builder()
                                         .addPlugin(LegacyMd5Plugin.create())
                                         .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
                                         .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
                                         .build();

Modifications

A new plugin which updates the configs
New APIs on ClientConfigBuilders
Interceptor that is same as HttpChecksumStage-legacyChecksum

Testing

Junits
One-off integs
A Test class was created with following APIs

    public void runAllTests() {

            createBucket();
            putObject();
            listObjects();
            getObject();
            deleteObjects();
            deleteBucket();
            System.out.println("All tests completed successfully!");
    }

Tested with minio version RELEASE.2025-01-18T00-31-37Z (commit-id=4b6eadbd80313711c01039bfa9a05167291a8c50) where the issue existed and all test passed , below config were tested

** With LegacyMd5Plugin **

        S3Client s3Client = S3Client.builder()
            .addPlugin(LegacyMd5Plugin.create())
            .endpointOverride(URI.create("http://127.0.0.1:9000"))
            .credentialsProvider(StaticCredentialsProvider.create(
                         AwsBasicCredentials.create("minioadmin", "minioadmin")))
                          .forcePathStyle(true) // Required for MinIO
                           .build();

** With LegacyMd5Plugin with ChecksumCalculation and ChecksumValidation as when required**

        S3Client s3Client = S3Client.builder()
            .addPlugin(LegacyMd5Plugin.create())
            .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
            .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)
            .endpointOverride(URI.create("http://127.0.0.1:9000"))
            .credentialsProvider(StaticCredentialsProvider.create(
                         AwsBasicCredentials.create("minioadmin", "minioadmin")))
                          .forcePathStyle(true) // Required for MinIO
                           .build();

** Also tested the above test cases and configutation with a real bucket in S3 and test passed **

Screenshots (if appropriate)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

License

I confirm that this pull request can be released under the Apache 2 license

…t got S3

L-Applin

architecture-tests fails because HttpChecksumUtils (intenal API in sdk-core) is used in another package (s3). We should either mark this utils class protected api or update the archunit store.

services/s3/src/main/java/software/amazon/awssdk/services/s3/LegacyMd5Plugin.java

zoewangg · 2025-04-25T16:30:14Z

services/s3/src/main/java/software/amazon/awssdk/services/s3/LegacyMd5Plugin.java

+    @Override
+    public void configureClient(SdkServiceClientConfiguration.Builder config) {
+        S3ServiceClientConfiguration.Builder s3Config = (S3ServiceClientConfiguration.Builder) config;
+        s3Config.responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED);


Should we not set it if responseChecksumValidation is already configured? Same for RequestChecksumCalculation (I missed it when I wrote the sample code😅)

Can you you please elaborate , the default values of RequestChecksumCalculation and ResponseChecksumValidation is WHEN_SUPPORTED thus this values will always have some values.

Can you please help me with a Scenario where this additional check might be helpful ?

On second thought, I believe we should not be doing:

s3Config.responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED); s3Config.requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED);

This is to maintain the single responsibility of the LegacyMd5Plugin, which is solely added to add the MD5 header.

The responseChecksumValidation and requestChecksumCalculation should instead be set on the client or in the environment variables. What do you think?

zoewangg · 2025-04-25T16:32:25Z

services/s3/src/main/java/software/amazon/awssdk/services/s3/LegacyMd5Plugin.java

+ *
+ * <p>Use this plugin only when you need to maintain compatibility with applications that depend on the
+ * legacy MD5 checksum behavior, particularly for operations that previously calculated MD5 checksums
+ * automatically.


Can we add sample code of how customers can configure it on the client with @snippet?

…egacyMd5Plugin.java Co-authored-by: Olivier L Applin <[email protected]>

…java-v2 into joviegas/legacy_md5_pluggin

joviegas · 2025-04-25T22:18:40Z

architecture-tests fails because HttpChecksumUtils (intenal API in sdk-core) is used in another package (s3). We should either mark this utils class protected api or update the archunit store.

Done , removed the HttpChecksumUtils apiand made a inline.

joviegas · 2025-04-25T22:21:46Z

architecture-tests fails because HttpChecksumUtils (intenal API in sdk-core) is used in another package (s3). We should either mark this utils class protected api or update the archunit store.

Done , I removed the HttpChecksumUtils api and made a inline. Lets see if it passes.

zoewangg · 2025-04-25T22:33:13Z

services/s3/src/main/java/software/amazon/awssdk/services/s3/LegacyMd5Plugin.java

+ *
+ * <p>This plugin configures the S3 client to:
+ * <ul>
+ *   <li>Set request checksum calculation to WHEN_REQUIRED mode, which calculates default checksums only when


Looks like we need to update this documentation since we removed configuration

zoewangg · 2025-04-25T22:34:38Z

services/s3/src/main/java/software/amazon/awssdk/services/s3/LegacyMd5Plugin.java

+ *
+ *  // For asynchronous S3 client
+ *  S3AsyncClient asyncClient = S3AsyncClient.builder()
+ *                                          .addPlugin(create())


LegacyMd5Plugin.create(). Can we configure the request checksum calculation and response checksum validation to when_required in the code snippet?

L-Applin · 2025-04-28T16:55:46Z

...t/java/software/amazon/awssdk/services/s3/plugin/legacymd5/LegacyMd5PluginOperationTest.java

Can we add a test cases with the plugin when user manually set md5 on the request body?

Added a test case both for putObject and deleteObjects

sonarqubecloud · 2025-04-28T20:32:41Z

Quality Gate failed

Failed conditions
64.1% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

steveloughran · 2025-04-30T09:27:52Z

thank you for this!

have you tried testing it against a third party store? google cloud is easy to set up & documented in the s3a thirdparty and qualification docs. Even if you only run it weekly, it'd restore my confidence in the SDK changes.

thanks

For rationale: aws/aws-sdk-java-v2#6055

joviegas · 2025-04-30T18:19:56Z

thank you for this!

have you tried testing it against a third party store? google cloud is easy to set up & documented in the s3a thirdparty and qualification docs. Even if you only run it weekly, it'd restore my confidence in the SDK changes.

thanks

HI Steve ,

I tested it with minio version RELEASE.2025-01-18T00-31-37Z where this issue was easily repeatable.
let me do a one-off test with google cloud , thanks for the reference
run it weekly will keep this in consideration and discuss about it (might take some time) .

For rationale: aws/aws-sdk-java-v2#6055

uwolfer · 2025-05-14T06:52:51Z

Has somebody already successfully tested this change against GCP? I have enabled the new plugin, but putObject still fails with a SignatureDoesNotMatch error while it works with the same code with 2.29.52.

You can also see some sample code in the related issue: #5987

joviegas · 2025-05-19T19:04:13Z

Hi @uwolfer ,
I have only tested with minio and it worked fine.

Also along with the pluggin could you please make sure ChecksumCalculation option are set as below

       S3Client s3Client = S3Client.builder()
            .addPlugin(LegacyMd5Plugin.create())
            .requestChecksumCalculation(RequestChecksumCalculation.WHEN_REQUIRED)
            .responseChecksumValidation(ResponseChecksumValidation.WHEN_REQUIRED)

If issue stil persist can you please check wire logs and check good and bad case Wire logs ?
I spent some time using gcp but it was giving me lot of authentication issues.
Can you please share some simple docs/link/steps which can help me to do a quick test with GCP.
@bhoradc did you get a chance to check it with GCP.

uwolfer · 2025-05-30T06:51:50Z

Also along with the pluggin could you please make sure ChecksumCalculation option are set as below

@joviegas Is it really the idea to disable optional checksum checks? If I remember correctly, it did even work without adding the new LegacyMd5Plugin plugin that way. But I would feel more comfortable if we could keep the checks as they were before 2.30.

Can you please share some simple docs/link/steps which can help me to do a quick test with GCP.

Here you can find a simple code snippet to test it against GCP (just insert the access key id and secret and pass some random content): #5987 (comment)

joviegas requested a review from a team as a code owner April 24, 2025 21:39

LegacyMd5Plugin which adds MD5 checksum for operations that require i…

aae6a51

…t got S3

joviegas force-pushed the joviegas/legacy_md5_pluggin branch from 32af381 to aae6a51 Compare April 24, 2025 22:54

joviegas changed the title ~~LegacyMd5Plugin which adds MD5 checksum for operations that require i…~~ Add LegacyMd5Plugin for MD5 checksum calculations in S3 operations requiring checksums Apr 24, 2025

joviegas added 2 commits April 24, 2025 17:52

LegacyMd5Plugin which adds MD5 checksum for operations that require i…

948760c

…t got S3

Merge branch 'master' into joviegas/legacy_md5_pluggin

a201cde

L-Applin requested changes Apr 25, 2025

View reviewed changes

services/s3/src/main/java/software/amazon/awssdk/services/s3/LegacyMd5Plugin.java Show resolved Hide resolved

zoewangg requested changes Apr 25, 2025

View reviewed changes

Update the comment

a7587ea

joviegas force-pushed the joviegas/legacy_md5_pluggin branch from 1a2bace to a7587ea Compare April 25, 2025 16:39

joviegas and others added 4 commits April 25, 2025 09:48

Update services/s3/src/main/java/software/amazon/awssdk/services/s3/L…

d5970e8

…egacyMd5Plugin.java Co-authored-by: Olivier L Applin <[email protected]>

Handled Oliviers commets

c718851

Merge branch 'joviegas/legacy_md5_pluggin' of github.com:aws/aws-sdk-…

afbe93a

…java-v2 into joviegas/legacy_md5_pluggin

Handled Zoes comment

d190de8

zoewangg reviewed Apr 25, 2025

View reviewed changes

joviegas added 2 commits April 28, 2025 06:39

Merge branch 'master' into joviegas/legacy_md5_pluggin

ad61cf3

updated Java docs for class

677fb24

joviegas requested review from zoewangg and L-Applin April 28, 2025 15:12

zoewangg approved these changes Apr 28, 2025

View reviewed changes

joviegas enabled auto-merge April 28, 2025 16:15

L-Applin reviewed Apr 28, 2025

View reviewed changes

Updated test case for user set md5 checksum

5a74563

L-Applin approved these changes Apr 28, 2025

View reviewed changes

Merge branch 'master' into joviegas/legacy_md5_pluggin

ed32910

joviegas added this pull request to the merge queue Apr 28, 2025

Merged via the queue into master with commit 4bf1f8f Apr 28, 2025
17 of 19 checks passed

stubz151 mentioned this pull request Apr 29, 2025

S3: Add LegacyMd5Plugin to S3 client builder apache/iceberg#12264

Open

wendigo added a commit to trinodb/trino that referenced this pull request Apr 30, 2025

Improve compatibility with S3-compliant storages

c932f96

For rationale: aws/aws-sdk-java-v2#6055

wendigo added a commit to trinodb/trino that referenced this pull request Apr 30, 2025

Improve compatibility with S3-compliant storages

a8ba2e2

For rationale: aws/aws-sdk-java-v2#6055

wendigo added a commit to trinodb/trino that referenced this pull request May 1, 2025

Improve compatibility with S3-compliant storages

ffc3d18

For rationale: aws/aws-sdk-java-v2#6055

wendigo added a commit to trinodb/trino that referenced this pull request May 1, 2025

Improve compatibility with S3-compliant storages

d16f83f

For rationale: aws/aws-sdk-java-v2#6055

joviegas deleted the joviegas/legacy_md5_pluggin branch May 29, 2025 15:05

Add LegacyMd5Plugin for MD5 checksum calculations in S3 operations requiring checksums #6055

Add LegacyMd5Plugin for MD5 checksum calculations in S3 operations requiring checksums #6055

Uh oh!

Conversation

joviegas commented Apr 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation and Context

Modifications

Testing

Screenshots (if appropriate)

Types of changes

License

Uh oh!

L-Applin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joviegas Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joviegas commented Apr 25, 2025

Uh oh!

joviegas commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joviegas Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 28, 2025

Quality Gate failed

Uh oh!

Uh oh!

steveloughran commented Apr 30, 2025

Uh oh!

joviegas commented Apr 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uwolfer commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joviegas commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uwolfer commented May 30, 2025

Uh oh!

Uh oh!

joviegas commented Apr 24, 2025 •

edited

Loading

joviegas Apr 25, 2025 •

edited

Loading

joviegas commented Apr 25, 2025 •

edited

Loading

joviegas Apr 28, 2025 •

edited

Loading

joviegas commented Apr 30, 2025 •

edited

Loading

uwolfer commented May 14, 2025 •

edited

Loading

joviegas commented May 19, 2025 •

edited

Loading