Performance test PVC with a cluster saturated with workspaces

[kylos101]

Is your feature request related to a problem? Please describe

We should test how a node behaves when it is full of workspaces using PVCs, where there is disk activity in the workspaces, and many workspaces are starting, running, and stopping.

The # of regular workspaces we decide to run as part of this test should be 125% of the peek volume we saw in our EU cluster over the last month.

Describe the behaviour you'd like

1. Start a regular workspace in the cluster using PVC on a single node.
2. Run loadgen, once to fill half of the cluster to capacity
3. Run loadgen
   Begin to stop the first loadgen run & start the second one with ~20 workspaces

Additional context

We haven't run with PVC snapshot or restore at scale, yet.

[sagor999]

It worked without issues.
Tested with loadgen with 20 workspaces, then with 30, then with 100. (20 and 30 with stop and start at the same time).

[kylos101]

@sagor999 why 100 workspaces, instead of 125% of our peek volume in the last month? 🤔 I would feel comfortable, even though it will cost to run the test, that we can handle that related volume. This way we're not doing for the first time in production. 😅

Can you show the start and stop graphs from the overview dashboard, so we can peek at what those shapes look like? For example, I don't assume they'll be much different than now...but am curious to confirm/compare.

Also, can you share how we're supposed to do related clean-up after testing in our loadgen readme.md? This will be good for the future.

Lastly, if you could add this to our project and set status, that would be 👌 , too! 🙇

[sagor999]

@kylos101 was concerned about the cost. But if you are ok with it, I can do loadgen with 700 workspaces. ok?

Here is the graph of workspace start up. It does look the same as regular workspaces to me.

[kylos101]

👋 @sagor999 thanks for sharing bud, I am okay with the cost 👍 👍 , especially it finds issues or mitigates risk. Could you do 900? We recently hit 830. 🚀 So long as you can delete snapshots after, we should be 🆗 .

[kylos101]

Hi @sagor999 , it looks like we have [an issue](https://github.com//issues/13353), sometimes, and, we're unsure if it's related to load.

For now, I recommend the following:

1. Close this issue, 12747
2. In [PVC] loadgen testing Pod can't mount Volume #13353, run a load test with a large number of workspaces, to see whether that stress causes trouble.

[kylos101]

As #13353 was resolved, I recommend proceeding with a load test of large scale. 👯

[jenting]

As #13353 was resolved, I recommend proceeding with a load test of large scale. 👯

Pavel asked me to load the test with 1000 regular workspaces. https://www.notion.so/gitpod/PVC-roll-out-plan-for-SaaS-b2fc4aa9a6304bd283263cdce008911d#d606efcba21a4cb4b4182f530273d154