Rate limit prebuilds

[jankeromnes]

Bug description

Today at around 13:24 UTC, there was a sudden jump in the number of prebuilds: Over about a minute, the repository https://gitlab.com/yo/gitlab triggered nearly 100 new prebuilds.

Unfortunately, this put excessive pressure on the US cluster, and in particular on the database, which lead to several workspaces not being able to start (for example: https://community.gitpod.io/t/django-pod-is-down/4541).

This incident lasted about 20 minutes and was logged here: https://www.gitpodstatus.com/incidents/ddyzghh3mfbm

Steps to reproduce

Instantly trigger 100 prebuilds

Expected behavior

There should be no outage, especially in the database

Example repository

No response

Anything else?

Maybe it would make sense to introduce a reasonable rate limit on prebuilds, for example per repository.

[JanKoehnlein]

/schedule

[AlexTugarev]

Looking into the old solution for now: https://github.com/gitpod-io/gitpod/blob/main/components/server/ee/src/prebuilds/prebuild-queue-maintainer.ts

I hope, @csweichel or @geropl find time to provide more details on this, please 🙏🏻

---

After thinking this through. I think the prebuild queue thingy is not the right way to solve this problem. Instead we should add a proper rate limit on projects, which equals to repos in a sense.

---

'Talked to Sven and we decide to try to solve it with an extra storage to the side of servers. That should be basically a queue and servers should pick a prebuild event when ready to execute. We assume that having a proper tech that already solves the queueing (such as redis) would also allow us to manage prebuilds properly, i.e. enforce rate limitation and implement throttling.

[JanKoehnlein]

I don't think this is actively worked on, and I also think this has to be specified and broken into smaller issues . Removing it from groundwork.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[geropl]

This just lead to an incident: https://app.incident.io/incidents/101
Thus added it to #7812

/cc @jldec