`task` bucket usage vs "directory" within a bucket

[DavidGOrtega]

We are generating a bucket with every task, however this approach has several outcomes and a bug:

• We are limited to the number of tasks
• Removal is tricky and maybe not desirable by the user. They might want to keep the tasks for a long time
• If your quota is depleted you can not delete tasks even to create new ones bug

A better approach would be having specified a bucket by the user or if not create the default .tpi bucket.

[0x2b3bfa0]

Limits

As per the research below, users can have at least 100 concurrent tasks running in any of the supported providers. Upon request, limits can be increased up to 250 per region in the worst case scenario.

Highly parallelizable tasks, like hyperparameter optimization, can use the parallelism argument to lauch up to thousands of machines with shared storage, without exceding any of these limits.

Moreover, the number of concurrent tasks is usually bound to the orchestration limits, not to the storage limits.

Storage

aws

By default, you can create up to 100 buckets in each of your AWS accounts. If you need additional buckets, you can increase your account bucket limit to a maximum of 1,000 buckets by submitting a service limit increase. There is no difference in performance whether you use many buckets or just a few.

az

Number of storage accounts per region per subscription, including standard, and premium storage accounts: 250.

gcp

There are no limits on the number of buckets you can create in Google Cloud Storage. — Jeff Terrace, Senior Software Engineer at Google, Google Cloud Storage.

k8s

There is no limit for the number of persistent volume claims beyond the provisioning limits of the underlying storage class.

Orchestration

aws

Auto Scaling groups per Region: 200 [...] To request an increase, use the Auto Scaling Limits form.

az

Maximum number of scale sets in a region: 2,500

gcp

Instance groups: Quota [...] This quota is per project. (empirically, defaults to 100)

k8s

There is no limit for the number of jobs that can be created in a cluster.

[0x2b3bfa0]

Deletion

Tasks have been designed to be deleted as soon as they finish or, rather, as soon as the user realizes that they have finished; e.g. the next morning. Even in moderately big teams, it would be unusual to have more than 100 concurrent tasks when following this approach.¹

Teams with more than ~20 data scientists will necessarily have to be backed by specialized DevOps engineers, who already have the ability of increasing those limits, and even overcoming them with workarounds.²

Footnotes

1. Tasks, not machines; as per https://github.com/iterative/terraform-provider-iterative/issues/299#issuecomment-979514335, the latter have much higher limits. ↩

2. In the worst case, many of these limits can be avoided by using multiple regions, accounts or other provider–specific partitions. ↩

[0x2b3bfa0]

Persistence

Task storage is not meant to replace persistent data/model storage and versioning tools like DVC. It's only meant to share state (e.g. checkpoints) between several short–lived machines.

Even DVC, which is a data–oriented tool, doesn't¹ include a mechanism to create (and much less delete) persistent storage resources. This is something that most organizations prefer to manage separately and in often disparate ways.

Because of the diversity and complexity involved in persistent storage provisioning, it would be risky to include such a feature as part of this project.

Logs

Likewise, task storage is not meant to replace log management platforms.² For the same reasons as data, log ingestion, storage and monitoring should be performed by means of specialized tools.

Responsibility

Infrastructure management tools have the responsibility of deleting all the resources they create, even when those resources are meant to exist for long periods of time. Creating long–lived resources implies providing a way of deleting them.³

Footnotes

1. At least, that's what the official documentation says. ↩

2. For example, Amazon CloudWatch, Azure Monitor or Google Cloud Logging; links courtesy of @casperdcl. ↩

3. Requiring ClickOps to delete automatically created resources is wrong on so many levels. Big organizations won't run a tool that exhibits such a behavior, and smaller ones shouldn't, even if they were willing to. ↩

[DavidGOrtega]

Here we go again!
I having serious issues today with this. I can not even imagine people adopting this after the issues they might have. Right now i do not have more buckets because I use this in the CI and we do not have an effective destroy in place.
I have requested more buckets to AWS and im still waiting approval! So Im totally locked.
Seriously I wont use something like this

[DavidGOrtega]

Even in moderately big teams, it would be unusual to have more than 100 concurrent tasks when following this approach.

Who says? I can show you cases where 100 is super small. As we have been discussing a team with 3 models and a matrix can launch more than those 100 in a breeze and we do not have a perfect destroy in the CI yet

[0x2b3bfa0]

As stated on #299 (comment), matrix use cases would benefit of parallelism and would only consume a single bucket and a single orchestration resource for all the created machines. See 0x2b3bfa0/cml-use-case-matrix-task for a rudimentary example.

[0x2b3bfa0]

Unfortunately, quota requests take time. In the meanwhile, #314 should be addressed in order to prevent the bug or, at least, to mitigate it. Let's demote this discussion to important after extracting the critical issue.

[0x2b3bfa0]

Comments on automatic deletion of task resources belong to #289