S3 Transfer Manager upload causes memory issues

[joernschumacher0001]

Describe the bug

Trying to upload a large amount of file (~50GB in about 400 files) with the TransferManager uses a huge amount of memory. The JVM size (as reported by top) greatly exceeds the max heap size configured at startup (we start with -mx6G so we can stay below 16GB).

We tried adjusting the available configurations of both transfer manager and async S3 client, with no success; this involved a lot of trial and error because it was not clear how they interact exactly.

• the TransferManager can be given an Executor, but then seems to immediately empty the requests into the asyncronous client?
• we limited the S3Client's maxConcurrency, but this didn't have to have any influence
• we did not find any configuration for total or individual connections' buffer size

Expected Behavior

The transfer manager could throttle requests towards the S3 client if too many failed requests occur, and/or the configuration should enable the configuration of a maximum total buffer.

Current Behavior

This results in many of the uploads to fail in the best case, in application shutdown (by the OS because of host RAM exhaustion) in the worst.

If the uploads fail, a CompletedDirectoryUpload is returned, correctly containing the failed files, and the memory situation recovers.

Outputting the CompletedDirectoryUpload.failedDownloads() result looks like this:

[FailedFileUpload(request=UploadFileRequest(putObjectRequest=PutObjectRequest(Bucket=<our-bucket>, Key=<target-path>, source=<local_file>, configuration=[<our-logging-listener>]), exception=software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Java heap space), [...]]

The configured TransferListener logs success and failure; the failure case looks like

software.amazon.awssdk.core.exception.SdkClientException: Failed to send the request: A callback has reported failure.
	at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
	at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:43)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.handleError(S3CrtResponseHandlerAdapter.java:127)
	at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onFinished(S3CrtResponseHandlerAdapter.java:93)
	at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onFinished(S3MetaRequestResponseHandlerNativeAdapter.java:24)

Our current workaround is to set the JVM memory low enough that the OS does not shut down the process, so we get returned the list as described; we can then retry single file which works fine.

Reproduction Steps

This is our setup:

var tm = S3TransferManager.builder()
   .s3Client(S3AsyncClient.crtBuilder()
      .credentialsProvider(awsCredentialsProvider)
      .maxConcurrency(10)
      .targetThroughputInGbps(0.6d)
      .build())
   .build();
tm.uploadDirectory(builder -> builder
            .bucket(ourBucket)
            .source(Path.of(ourDirectory))
            .s3Prefix(s3DirectoryKey)
            .uploadFileRequestTransformer(transformer -> transformer.addTransferListener(loggingListener))
        )
}

The loggingListener really only logs success and failure:

class LoggingTransferListener implements TransferListener {
        @Override
        public void transferComplete(final Context.TransferComplete context) {
            logService.info(log, "transfer complete");
        }

        @Override
        public void transferFailed(final Context.TransferFailed context) {
            logService.error(log, "transfer failed", context.exception());
        }
}

We're assuming the main issue is the amount of data.

Possible Solution

• allow overall available memory configuration
• size the individual connections' buffer dynamically according to available or configured memory

Additional Information/Context

Our use case is persisting a Kafka Streams state store, to give an idea of how the data to be stored is structured. Total size is about 20-25GB across ~400 files as mentioned in the bug description.

AWS Java SDK version used

2.19.21

JDK version used

17.0.6

Operating System and version

Linux 4.14.301-224.520.amzn2.x86_64

[zoewangg]

Hi @joernschumacher0001, have you tried lowering the initialReadBufferSizeInBytes?
https://github.com/aws/aws-sdk-java-v2/blob/master/services/s3/src/main/java/software/amazon/awssdk/services/s3/S3CrtAsyncClientBuilder.java#L149 (edit: nvm, the config only applies to download)

It seems the issue is that the SDK is not uploading as fast as reading from file. Lowering the concurrency and targetThroughputInGbps would actually make it worse. What's your network bandwidth? Can you try increasing targetThroughputInGbps?

[joernschumacher0001]

Hello @zoewangg,
thank you for the quick reply.

That's what I assume as well; my expected behavior would be for the client to pause reading from file eventually when it realizes it can't get rid of the data.

We initially did not use either targetThroughput nor maxConcurrency.

We put in the targetThoughput because when we didn't use it, the client network became overflooded and the host eventually shutdown because it couldn't answer health queries from the ELB anymore (it pretty much did to the network what it does to memory now) - so we'll need to keep this configuration.
Our baseline bandwidth is 0.781 Gbps, burst 12.5 (which doesn't cover enough to upload all the files).

We did remove the maxConcurrency, but it didn't seem to change anything.

Some more points I realized could be relevant:

• we started using the crt client because when we used the default AsyncS3Client.builder(), we ran out of connections, and because it allowed us to configure the targetThroughput in the first place. That makes me assume that the memory issue itself is caused by the CRT client; but either way, our expectation is that the client should keep a closer eye on their resource usage (network and memory), and the TransferManager should take care of the underlying client's issues (so essentially, from our point of view this is two bugs).
• we are running our service in a docker container (which may well be relevant to how the available memory is determined)

Note: I edited the original problem description, we're actually trying to transfer 50GB of data; but if I remember correctly, it already happened on ~25GB (the difference comes from originally occurring on our develop environment and us now running it on the integration environment).

Another note: we did manage to work around our problem by using the "manual" retry, so our system does run fine now, although it did take us a few weeks to finalize stabilizing it. I appreciate you helping us trying to find a configuration to fix our issue, but our main reason for posting this is to make you aware of this behavior.

[zoewangg]

Thank you for the additional information! We will investigate it.

[TingDaoK]

Hi @joernschumacher0001, just for your info

1. we limited the S3Client's maxConcurrency, but this didn't have to have any influence

I saw in your code snap, it set to 10. We calculated the max connections based on your targetThroughputInGbps, the math is basically ceil(targetThroughputInGbps/4) * 10. So, in your configuration, without your override of max connection number, it will be set to 10, which means your override doesn't really do anything here.

[graebm]

We've reproduced the bug and identified the immediate cause. The Transfer Client is eagerly buffering the first 16MB of each file, even before any of those files start sending. If there are 400 files, that means an immediate allocation of 6+ GB.

We're changing this buffering to be more lazy. Since the underlying HTTP layer only works on a few files at a time, this will hopefully address the issue.

[mikomatic]

We also have this usecase uploading a folder (AWS console reports 23881 objects, 24.9 GB). We encounter the same issues of memory management (our pod kept getting OOMkilled).

We managed to make it work by raising the memory in our pod.

Our client also use CRT with default config

  public S3TransferManager s3TransferManager() throws URISyntaxException {
    String s3URL = this.appProperties.getS3().getUrl();
    String s3Region = this.appProperties.getS3().getRegion();

    S3AsyncClient s3AsyncClient =
        S3AsyncClient.crtBuilder()
            .region(Region.of(s3Region))
            .endpointOverride(new URI(s3URL))
            .build();

    return S3TransferManager.builder().s3Client(s3AsyncClient).build();
  }

Our method

  public CompletableFuture<CompletedDirectoryUpload> uploadFolder(
      Path folderToUpload, String bucket, String keyPrefix) {

    log.debug(
        "Uploading folder {} to bucket [{}] with key [{}] ", folderToUpload, bucket, keyPrefix);

    try {
      return s3TransferManager
          .uploadDirectory(
              builder -> builder.bucket(bucket).s3Prefix(keyPrefix).source(folderToUpload))
          .completionFuture();

    } catch (AwsServiceException | SdkClientException e) {
      throw new S3Exception("Unable to upload files to S3", e);
    }
  }

I'm commenting also to notify an additional behaviour we are seeing, there's seems to be a memory leak also in the uploadDirectory.

Our process (download, unzip, upload) takes a few minutes to complete, the memory never goes down ( prometheus data on a fresh pod)

I don't exclude that the memory leak comes from our application code, but it only happens when calling s3TransferManager.uploadDirectory .

[dmurphy-ambra]

I've been monitoring this issue, as we are considering switching back to the S3CrtAsyncClient from the vanilla S3AsyncClient. The fix seems to have been completed sometime ago - is there a timeline for when it will be merged and release?