Monitor the update-downloads script #1797
Comments
sgrif
added a commit
to sgrif/crates.io
that referenced
this issue
Aug 7, 2019
This replaces the `update-downloads` binary with a background job, and a binary that is used to queue up a given job, which we will run from Heroku scheduler. This accomplishes 2 things: - It makes it easier to write tasks that need to run periodically (e.g. cleaning up stale rate limit buckets), since we don't need to create a new standalone binary. - `update_downloads` and any future recurring tasks will automatically get monitoring if they fail, since we are already monitoring for background jobs not being successfully run. Right now the intent is to have `enqueue-job update_downloads` get run periodically by Heroku scheudler (and a similar scheduled task for any future tasks that are added). Once swirl gains the ability to schedule jobs to be run at arbitrary points in the future, we could instead have these jobs re-queue themselves once they complete, and have the cron task just look to see if any job is queued for each given type, queuing it if not. That would have a bit less boilerplate, but a lot more complexity. Fixes rust-lang#1797.
sgrif
added a commit
to sgrif/crates.io
that referenced
this issue
Aug 7, 2019
This replaces the `update-downloads` binary with a background job, and a binary that is used to queue up a given job, which we will run from Heroku scheduler. This accomplishes 2 things: - It makes it easier to write tasks that need to run periodically (e.g. cleaning up stale rate limit buckets), since we don't need to create a new standalone binary. - `update_downloads` and any future recurring tasks will automatically get monitoring if they fail, since we are already monitoring for background jobs not being successfully run. Right now the intent is to have `enqueue-job update_downloads` get run periodically by Heroku scheudler (and a similar scheduled task for any future tasks that are added). Once swirl gains the ability to schedule jobs to be run at arbitrary points in the future, we could instead have these jobs re-queue themselves once they complete, and have the cron task just look to see if any job is queued for each given type, queuing it if not. That would have a bit less boilerplate, but a lot more complexity. Fixes rust-lang#1797.
bors
added a commit
that referenced
this issue
Aug 7, 2019
…jtgeibel Move `update-downloads` to a background job This replaces the `update-downloads` binary with a background job, and a binary that is used to queue up a given job, which we will run from Heroku scheduler. This accomplishes 2 things: - It makes it easier to write tasks that need to run periodically (e.g. cleaning up stale rate limit buckets), since we don't need to create a new standalone binary. - `update_downloads` and any future recurring tasks will automatically get monitoring if they fail, since we are already monitoring for background jobs not being successfully run. Right now the intent is to have `enqueue-job update_downloads` get run periodically by Heroku scheudler (and a similar scheduled task for any future tasks that are added). Once swirl gains the ability to schedule jobs to be run at arbitrary points in the future, we could instead have these jobs re-queue themselves once they complete, and have the cron task just look to see if any job is queued for each given type, queuing it if not. That would have a bit less boilerplate, but a lot more complexity. Fixes #1797.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We've had multiple cases where the update-downloads script was erroring and no one caught it until someone mentioned their crate's download counts weren't updating days later.
Someone should get paged if this happens, which we could detect by:
version_downloadsrecords withprocessed = 'f'getting over some thresholdI'm not sure which would fit with our existing paging infrastructure best.
The text was updated successfully, but these errors were encountered: