Slow Storage uploads and high CPU usage

[cbraynor]

I'm seeing very slow uploads when sending large numbers of medium-sized files to storage - in the order of it taking 10x the time compared to uploading to S3. I also see excessive CPU usage during this time.

This Gist is a repro in the form of a side-by-side comparison uploading the same set of 750 medium-sized random text files to S3 using knox and to GCS using gcloud-node

gsutil can upload the same number of files just as quickly as anything to S3, so I'm pretty sure it's not the service itself.

I've tried limiting the request module's default maxSockets to 5 or 10 for the request pool but that didn't seem to help. I have a hunch it's the sheer number of outstanding requests or streams that's causing node to spin its wheels and that maybe some form of global queue could fix it, but I haven't been able to validate that yet.

Any help getting this into a usable state would be much appreciated

[stephenplusplus]

Thanks for the detailed breakdown. For me, the gcloud uploads have completed between 88-90 seconds consistently. However, there are two things that are on by default with every write stream which could be contributing towards higher memory usage and slower speeds overall:

1. Data integrity - a hash is progressively built during an upload that is compared against the hash returned from the API when the upload is complete
2. Resumable - writes to a tmp file to keep track of the state of an upload

Shutting these off improved my speeds from 88-90 to 65-67. I'm not sure if the others (knox or gsutil) use these techniques as well.

[cbraynor]

Ok, so it looks like you're right and a lot of the issue is it being CPU-bound computing the hash - I'm testing on multi-core machines that don't have huge individual processors and I get a 5x improvement turning off the validate functionality. It's still not as fast as the knox implementation but it certainly is a good start for now

[ryanseys]

Should we consider turning validation to false by default?

[jgeewax]

I think that's an issue for discussion.

I'd also like to get an idea of what gsutil is doing and how it compares from a performance standpoint. If the only differences are runtime and validation, then I think our work is done and we just need to make a call about whether to validate hashes by default...

[dhermes]

@jgeewax gsutil uses the hashlib.md5 (standard library) and a third-party crcmod which uses a C extension for speedups.

Also, in terms of turning off checks, @thobrla filed googleapis/google-cloud-python#547 (via @silvolu) with us and said:

We should allow to override the configuration for the checks (e.g. use only
crc32c, do not perform checks), but it should be strongly discouraged.

[ryanseys]

It looks to me like gsutil is hashing the entire file at the end, where as we hash it as the contents comes in. You would think that doing it at the end would have worse performance, but doing it all at the end might reduce thrashing and reduce our CPUs context switching.

[jgeewax]

Interesting... Might be worth a benchmark...?

[ryanseys]

I ran on my machine and didn't see a huge spike in CPU (~20% max) but certainly was more time consuming to upload with default settings (validation and resumable is on). I'm running on S3 region us-east-1 and I'm located in Ottawa (mid-east Canada). Seems S3 is really sporadic with performance, as you can see by the fluctuation in numbers with no change in settings. GCS seems a little more stable in that regard.
Here's my numbers:

With default settings:

Run 1:

Finished uploading to s3 in 47.792 seconds
Finished uploading to GCS in 91.161 seconds

Run 2:

Finished uploading to s3 in 41.566 seconds
Finished uploading to GCS in 92.455 seconds

Run 3:

Finished uploading to s3 in 47.247 seconds
Finished uploading to GCS in 93.156 seconds

Without validation or resumable GCS:

Run 1:

Finished uploading to s3 in 39.286 seconds
Finished uploading to GCS in 54.476 seconds

Run 2:

Finished uploading to s3 in 57.732 seconds
Finished uploading to GCS in 56.977 seconds

Run 3:

Finished uploading to s3 in 42.917 seconds
Finished uploading to GCS in 52.861 seconds

I haven't tried with gsutil yet or moving hashing to run at the end.

[jgeewax]

Do we have numbers comparing gsutil to GCS versus gcloud-node to GCS?

[ryanseys]

I'm working on that now.

[ryanseys]

So I used the following command to upload the directory of pre-extracted files to a bucket on GCS using gsutil. The -m option stands for multi-threading or something and allows the uploads to occur in parallel, greatly speeding up the upload time:

$ time gsutil -m cp -r tmp gs://my-test-bucket/my-test-folder

Results:
22.04s user 2.93s system 71% cpu 34.866 total
21.00s user 2.99s system 67% cpu 35.477 total
21.66s user 2.92s system 70% cpu 35.069 total

[thobrla]

When possible, gsutil hashes on the fly. I think there's only one case when using the boto library (which is not the default for interacting with GCS) where it hashes at the end.