Blob.download_to_filename eating lots of RAM

[tweeter0830]

I have a large ~4 GB binary file on GCS. When I use blob.download_to_filename, my RAM usage goes increases by 3 GB temporarily. Thus behavior goes away if I explicitly set a chunk size for the blob.

This seems like bad default behavior.

[dhermes]

@lukesneeringer @jonparrott WDYT about modifying the default? Also, IIUC if we use requests / urllib3 by default we can still stream the entire payload (i.e. no chunking) into a file without having to put the whole thing in RAM first.

[theacodes]

Seems reasonable to me. A default of a few megabytes hopefully won't hurt performance for small files. Can you do some benchmarks?

[dhermes]

Fixed in #3713

[eddie-scio]

Resurrecting this -- would you expect that the streaming solution in #3713 wouldn't work if the file were a tmpfs volume? That's what I'm seeing, I think I still need to use chunk_size to avoid memory blowing up.