Simplify dartdoc storage

[isoos]

Right now we upload all the generated files separately to the bucket and serve / proxy them from there. Uploading them one-by-one takes a long time, way more than generating them.

I'm wondering if it would be less wasteful on resources if we were uploading only the archive, and when the dartdoc service gets a request to something inside, it would use a disk-based cache to serve these (and download the package archive if it is not in the cache).

@jakobr-google wdyt?

[jakobr-google]

I don't know how much local storage we have available, but it sound like it'd be worth exploring.

It would mean that first access to a package's doc would be slower, potentially across all of the (max 8) instances, though, so I'm not sure how well it'll perform.

[isoos]

We have 8 instance only because doc generation (and upload) takes a long time. Once it is done, it will go back to 2 instances.

We could also pre-fetch the latest archives of the most popular packages, making their access fast.

[isoos]

Based on a random sampling, I think the documentation of the latest stable versions compressed should be around 1-1.4 GB, but uncompressed size would be around 10-15 GB. However the access would not always on the latest, as cross-documentation links are versioned to the resolved package version.

The default instance that we are using has ~4 GB free disk space for our use. We can request more resources, but it may be cheaper to request memory and/or use memcache more aggressively for this.

Before going ahead with the actual implementation of the local-disk-based serving, I think I would like to do a bit of tracking first:

• storing the compressed and uncompressed size of the archive
• track the network traffic with the current direct-to-bucket requests
• emulate as if we were using an LRU-based local disk cache: what is the new disk and network usage

/rfc @kevmoo @sortie

[kevmoo]

Leaving it as an engineering decision. I support measuring first.

Have you looked at: https://cloud.google.com/storage/docs/best-practices#uploading

Also: I wonder if "shelling out" to gsutil would be faster – something to investigate?

Or maybe create a gRPC client? https://github.com/googleapis/googleapis/tree/master/google/storagetransfer/v1

[isoos]

Have you looked at: https://cloud.google.com/storage/docs/best-practices#uploading
Also: I wonder if "shelling out" to gsutil would be faster – something to investigate?

My (still not in-depth) search shows the following:

• there is no bulk upload api,
• gsutil uses the same api endpoints as we do,
• "storage transfer" is a service for transferring between buckets (e.g. from AWS S3, or between Goole Buckets)

There seems to be no quick win here. We could increase the concurrency of the uploads, but I think we have bigger gains if the archive-only approach works.

[kevmoo]

ack

…

On Mon, Jun 4, 2018 at 2:02 AM István Soós ***@***.***> wrote: Have you looked at: https://cloud.google.com/storage/docs/best-practices#uploading Also: I wonder if "shelling out" to gsutil would be faster – something to investigate? My (still not in-depth) search shows the following: - there is no bulk upload api, - gsutil uses the same api endpoints as we do, - "storage transfer" is a service for transferring between buckets (e.g. from AWS S3, or between Goole Buckets) There seems to be no quick win here. We could increase the concurrency of the uploads, but I think we have bigger gains if the archive-only approach works. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1333 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABCisMUq1Cdj16cAQJ8nhlFX9sbr6KKks5t5PeygaJpZM4US984> .

[isoos]

#1342 should provide reasonable data on the requirements, but with a bit more experimenting I fear that real decision point will be the latency of downloading the package.tar.gz and uncompressing it to local disk.

Maybe this should be decided per-PackageVersion:

• the latest or the recently published versions will get accessed more likely: they would be the same as today (per-file + archived uploads)
• older versions, especially the non-latest that are over a year old could be storing only the package.tar.gz, and when accessed, pay the latency hit

[sortie]

Some ideas without having looked much into the problem: Is there a way to upload an archive and ask the remote to extract it? Could we ask the cloud people a bulk upload option? Can we upload files individually but much more concurrently? Can we have a background process download the uploaded archives and convert them to individual files, without blocking the initial upload?

[isoos]

Is there a way to upload an archive and ask the remote to extract it? Could we ask the cloud people a bulk upload option?

I am not aware of anything like that, but would be happy if it would exists.

Can we upload files individually but much more concurrently?

When we have a new version deploy, we have ~16 process uploading files with concurrency=4 each. We can easily increase that concurrency, not sure what the effect would be in the storage bucket.

Can we have a background process download the uploaded archives and convert them to individual files, without blocking the initial upload?

That wouldn't save us much, as with the current structure, we don't make the new version active until all of the files were not uploaded. It would shift when the actual upload happens, but wouldn't unblock anything else.

[kevmoo]

You could always use triggers and pub/sub to schedule an extract of the archive after upload. More plumbing, but at least it could be done in the background...

[isoos]

Wow, Github's edit the comment feature allowed me to edit @kevmoo's comment, and it did in a way that I didn't realize it. Anyway, the reply to it:

You could always use triggers and pub/sub to schedule an extract of the archive after upload.

I don't think we win anything by it: the serving of those assets starts only after the files are extracted and uploaded again, and the process just adds more latency with the pub/sub queue handling.

The clear win is when we don't need to upload the individual files at all, only the archive.

[isoos]

Tracking was running for almost a day, and it revealed some interesting numbers already:

If we were to store and extract all archives locally, it would require at least 5 GB of disk space at its (current) peak, and we could be storing ~500 package version archives locally at any time. It is not tracked in an exact way, but based on the fluctuation of the size requirements, there is a large turnover of archives, as if there were a few requests for a certain version, and then that version is not requested anymore.

If we were to store only the obsolete version archives (non-latest, older than two years), it would be much less package at a time (~40) and much less disk space (current samples topped out at ~1 GB), but the turnover is there too.

It not clear if/how search engine optimisation in #1368 would affect these numbers.

[isoos]

It looks like that the load varies a bit, the numbers topped out this week:

• 15 GB disk space for all archives locally (~1800 at the same time)
• 2GB if only the archived ones (~100 at the same time)