TransferManager downloadDirectory woes

[dprestegard]

Hello!

I'm trying to use the v2 Java SDK's TransferManager to download a prefix recursively. In other words, I want to mirror the behavior of the --recursive flag on the AWS CLI. I'm following the SDK example here:

https://github.com/aws/aws-sdk-java-v2/blob/master/services-custom/s3-transfer-manager/README.md#download-s3-objects-within-the-same-bucket-to-a-local-directory

When I try to do this I get an exception like this.

Exception in thread "Thread-14" Exception java.io.UncheckedIOException: java.nio.file.FileSystemException: /var/tmp/getdir2 Is a directory

Here's the implementation.

    log.debug("Building async s3 client");
    S3AsyncClient s3AsyncClient = S3AsyncClient.crtBuilder()
            .credentialsProvider(DefaultCredentialsProvider.create())
            .region(Region.of(region))
            .targetThroughputInGbps(targetThroughputInGbps)
            .minimumPartSizeInBytes(minimumPartSizeInMB * MB)
            .build();

    log.debug("Building v2 Transfer Manager");
    S3TransferManager transferManager = S3TransferManager.builder()
            .s3Client(s3AsyncClient)
            .build();

    log.debug("Starting download");
    downloadObjectsToDirectory(transferManager, directory, bucket, prefix);

.......


  private void downloadObjectsToDirectory(S3TransferManager transferManager, String directory, String bucket, String aPrefix) {
    DownloadDirectoryRequest downloadDirectoryRequest = DownloadDirectoryRequest.builder()
            .destination(Paths.get(directory))
            .bucket(bucket)
            // only download objects with specified prefix
            .listObjectsV2RequestTransformer(l -> l.prefix(aPrefix))
            .build();

    DirectoryDownload directoryDownload = transferManager.downloadDirectory(downloadDirectoryRequest);

    // Wait for the transfer to complete
    CompletedDirectoryDownload completedDirectoryDownload = directoryDownload.completionFuture().join();

    // Print out any failed downloads
    completedDirectoryDownload.failedTransfers().forEach(fail ->
            log.error("Object [{}] failed to transfer", fail.toString()));

    Integer failedTransfers = completedDirectoryDownload.failedTransfers().size();
    if (failedTransfers > 0) {
      throw new S3GetDirException("Failed to download one or more files");
    }
  }

The prefix in this example is structured like this:

s3://bucket/prefix/
s3://bucket/prefix/data/
s3://bucket/prefix/artifacts/

and I want to do the equivalent of

aws s3 cp s3://bucket/prefix/ /var/tmp/getdir2/ --recursive

which would download everything in bucket/prefix/ - files and virtual folders included.

What am I missing>?

[debora-ito]

The exceptions are associated with the objects that were created as "folders".

S3 does not have the concept of folders, everything is an object, and a "folder" is just an object of size 0 and a separator, like prefix/. If you call ListObjects you'll see the objects with size 0 are probably the ones you are seeing in the exceptions.

That said, if those "folders" listed in the exceptions contain any objects, they should be downloaded successfully by TransferManager.downloadDirectory().

Let us know if this helps.

[dprestegard]

Thanks for the reply, @debora-ito :)

Indeed, I'm aware of how S3 handles "folders" - thanks for the context.

Interestingly, the exception is actually pointing to the folder on my local filesystem where I want the prefix on s3 to be downloaded to. In other words, the exceptions do NOT mention any objects / virtual folders in S3:

Exception in thread "Thread-5" java.io.UncheckedIOException: java.nio.file.FileSystemException: /var/tmp/getdir2: Is a directory
--
  | at software.amazon.awssdk.utils.FunctionalUtils.asRuntimeException(FunctionalUtils.java:180)
  | at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:110)
  | at software.amazon.awssdk.utils.FunctionalUtils.invokeSafely(FunctionalUtils.java:136)
  | at software.amazon.awssdk.core.internal.async.FileAsyncResponseTransformer.onStream(FileAsyncResponseTransformer.java:113)
  | at software.amazon.awssdk.core.async.listener.AsyncResponseTransformerListener$NotifyingAsyncResponseTransformer.onStream(AsyncResponseTransformerListener.java:93)
  | at software.amazon.awssdk.core.async.listener.AsyncResponseTransformerListener$NotifyingAsyncResponseTransformer.onStream(AsyncResponseTransformerListener.java:93)
  | at software.amazon.awssdk.core.internal.http.async.AsyncStreamingResponseHandler.onStream(AsyncStreamingResponseHandler.java:63)
  | at software.amazon.awssdk.core.internal.http.IdempotentAsyncResponseHandler.onStream(IdempotentAsyncResponseHandler.java:108)
  | at software.amazon.awssdk.core.internal.http.async.CombinedResponseAsyncHttpResponseHandler.onStream(CombinedResponseAsyncHttpResponseHandler.java:85)
  | at software.amazon.awssdk.core.internal.http.async.AsyncAfterTransmissionInterceptorCallingResponseHandler.onStream(AsyncAfterTransmissionInterceptorCallingResponseHandler.java:86)
  | at software.amazon.awssdk.services.s3.internal.crt.S3CrtResponseHandlerAdapter.onResponseHeaders(S3CrtResponseHandlerAdapter.java:62)
  | at software.amazon.awssdk.crt.s3.S3MetaRequestResponseHandlerNativeAdapter.onResponseHeaders(S3MetaRequestResponseHandlerNativeAdapter.java:28)
  | Caused by: java.nio.file.FileSystemException: /var/tmp/getdir2: Is a directory
  | at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
  | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
  | at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
  | at java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:200)
  | at java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:259)
  | at java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:323)
  | at software.amazon.awssdk.core.internal.async.FileAsyncResponseTransformer.createChannel(FileAsyncResponseTransformer.java:85)
  | at software.amazon.awssdk.core.internal.async.FileAsyncResponseTransformer.lambda$onStream$2(FileAsyncResponseTransformer.java:113)
  | at software.amazon.awssdk.utils.FunctionalUtils.lambda$safeSupplier$4(FunctionalUtils.java:108)
  | ... 10 more

In any case, all "folders" matching the specified prefix do in fact contain objects, so everything should download as I understand it.. yet nothing does.

I originally started with a more complex case of nested folders, but have made things even simpler. I want to download this:

s3://bucket/prefix/object_a
s3://bucket/prefix/object_b

by specifying prefix/ as the prefix to download.

This fails in the same way, with a FileSystemException complaining that my destination folder Is a directory. I've tried pre-creating this folder and NOT pre-creating this folder and I get the same exception

[debora-ito]

Got it, thank you for the clarification.

I tried to repro locally, here's what I found so far:

(1) I can reproduce the /var/tmp/getdir2: Is a directory error with this use case:

s3://bucket/prefix/object_a
s3://bucket/prefix/object_b

by specifying prefix/ as the prefix to download.

In my case, both object_a and object_b were downloaded successfully despite the error.
We'll look into why the exception is thrown.

(2) I tried more complex cases of nested folders but could not reproduce. Can you give more details on the complex cases?
An example of the object tree and a self-contained repro case with the exact prefix used would be ideal.

Which version of the SDK are you using?
And which JDK?