Performance issues with AWS Smithy HTTP Client stack

[russfellows]

Describe the feature

What: Performance limitations

When implementing the AWS S3 Rust crate for S3 access on version 1.103+ (aws-sdk-s3
v1.104.0), there are performance limitations.

Proof of Issue

How do we prove a negative? By comparing to something else.

Test case A

Using the MinIO warp s3 testing tool, I have achieved line rate read (get) speed, on a single client over a single 100 Gb NIC. Specifically, I can repeatedly achieve 11.5 GiB/s and up to 12 GiB/s with a specific setup to local S3 compatible storage.

Test case B

Using the AWS sdk Rust stack (directly utilizing the AWS S3 crate version 1.103 and later), we are unable to achieve greater than 5.5 GiB read (get) speeds. This is using the IDENTICAL client system, IDENTICAL storage system, IDENTICAL network, etc. etc.

Details

After investigations using flame-graph to investigate, it is apparent that there are issues, which appear to occur in the AWS Smithy HTTP Client stack. There are few public options in this crate, but even after forking a copy and modifying private functions and structs to become public for modification, full performance was not achieved.

Use Case

The use case is ANY S3 use case where high-performance is desired.

Proposed Solution

Fix the HTTP client software stack.

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

A note for the community

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue, please leave a comment

[russfellows]

Note: this is related to the bug just filed with the AWS-S3-C library: [aws-c-s3 crt issue 562] (awslabs/aws-c-s3#562)

[landonxjames]

The SDK should be capable of those speeds, barring any recent performance regressions. We have some benchmarks from the development of the S3 Transfer Manager showing it nearly maxing out the NIC of a beefy EC2 instance. Achieving that level of throughput requires carefully managing the concurrency of your application. This is difficult, but we do have a guide to help.

If you are mostly concerned with S3 performance using the S3 Transfer Manager is going to be your best bet to avoid having to manage all of that concurrency on your own. We would be happy to take feature requests in that repo if you find anything you need is missing.

If you want to achieve that level of throughput with the low level SDK client we will need some more information to help you debug. The first questions that come to mind are:

• How are you managing the concurrency of your application currently?
• What is your usage pattern? Are your getting a lot of small files in parallel, one big file in a single request, or using parallel ranged gets?
• Would it be possible to share your flamegraphs, or at least let us know which operations are taking up a lot of time?

[russfellows]

Landon, can you provides some specifics on what constitutes a "beefy EC2 instance"?

Thanks, but the he help guide will not work in my case, and also, for several reasons the S3 Transfer Manager is not applicable.

First, and most importantly, this is for GENERIC S3 storage, which means this code is expected to work against a variety of S3 storage, including but not limited to AWS' S3 storage. This library is being tested against MANY other S3 compatible targets.

For reference, I am using the s3dlio library, which is publicly available in a github repo: at s3dlio Github . As you will see, this is a Rust library built using your SDKs.

For performance comparison I am using the MinIO warp tool, also readily available on Github MinIO Warp

To answer your questions:
Q1) How am I managing concurrency?
A1) It is being managed via tunable parameters that control the number of async threads open. I am using Async I/O with Tokio in Rust.

Q2) What is the usage pattern?
A2) It is whatever the tester wants, and in the specific case I am referencing, it is all 8 MiB objects, which were created for the specific purpose of testing performance. But, the s3dlio cli allows you to create any number and size of objects.

Q3) Can I share my flamegraphs?
A3) Yes, I will share everything I have, including flamegraphs, the commands I ran, screenshots, etc. etc.

First, below I have provided 4 images. Two are the command line invocations, and two are nmon charts while a test was running. For comparison, I show the results of warp testing with 8 MiB objects. I then show running the same test with s3dlio, which is built using MANY AWS Rust crate SDKs. The analysis and flamegraphs all show waiting on the http client code in the AWS stack. Nmon confirms this, as well over 50% of the time is spent in wait. Also it has higher CPU and system usage that when running Warp. Of course when using s3dlio, I used the exact same size of objects, 8 MiB.

Again, you can reproduce this in your own environment by using Warp from MinIO, and s3dlio, which is available via the link I already provided.

So, below are these 4 images:

Next: Here are 3 flamegraphs that I produced. You can reproduce these in your environment as well, since the test cases used to generate these are part of the s3dlio project in github. Look under the "examples" directory for the Rust tests that generate these:

[landonxjames]

Landon, can you provides some specifics on what constitutes a "beefy EC2 instance"?

For our testing we used a c6in.32xlarge instance which has 2 100Gib/s NICs if I remember correctly. We were also able to patch Hyper to bind to two NICs and max out both of them (although this work hasn't made it into upstream Hyper and is not publicly available today).

First, and most importantly, this is for GENERIC S3 storage, which means this code is expected to work against a variety of S3 storage, including but not limited to AWS' S3 storage. This library is being tested against MANY other S3 compatible targets.

I am afraid that we do not officially support any other S3 compatible storage solutions and won't be doing any work to optimize for those use cases.

In the flamegraphs above I am not seeing anything terribly anomalous. The long poll appears to be actually performing the operation (signing, serializing, deserializing, etc.) and waiting on a network response. There are potentially ways you could speed that up: disabling checksums if you don't need them, split the individual file downloads into smaller ranged GETs to limit the amount of time waiting for the network to respond.

With respect to performance, we will take a look at improving the performance of individual requests, but any individual request is unlikely to max out a NIC. There is not much we can do to improve concurrency since that must be managed at the application level. If you are looking for inspiration on how to tune your concurrency for S3, the Scheduler struct is the entry point for how we do that in the transfer manager.