Switch gitpod team (eating your own dog food) to using PVC - regular workspaces only

[sagor999]

[sagor999]

@kylos101 I would like to switch gitpodders to using PVC feature on Monday (June 27th)
That would be only for new workspaces. (will not apply to prebuilds)

[aledbf]

I would like to switch gitpodders to using PVC feature on Monday (June 27th)

Not in SaaS right?

[sagor999]

In SaaS indeed. I assume you have objections? @aledbf
That would enable the part where when you start a new workspace, it will use PVC and load any prior data from GCS.
But not enabled for prebuilds, since that PR is not merged yet.

[aledbf]

In SaaS indeed. I assume you have objections? @aledbf

I think it is too risky to do it that way. I think it would be better to use a workspace class, meaning a separated node pool. Then, in case of any issue, the blast radius is limited to only a few nodes running only PVC workspaces from the team.

The problem with that idea is that we need to wait for Thomas work on workspaces classes :(

[sagor999]

Not entirely agree on it being too risky.
If anything, there is much less dependence on ws-daemon with PVC approach. So if anything, it should be more reliable, considering that ws-daemon crashes often.
Waiting for workspace classes will delay PVC roll out even longer.

[kylos101]

Removing from scheduled work for now (as you and JenTing are largely testing in ephemeral clusters, and will test later, probably gen61, with dedicated nodes).

[kylos101]

👋 @sagor999 ,

We're really close, just a few hurdles, I think, till we can ship reliably with PVC! 🙌

What work remains before we can dog food PVC with regular workspaces? I'm aware of the below. Is there anything missing or that I shared incorrectly? I ask so that we can leverage workspace teammates to help deliver and ship as part of gen64. 🤝

Loadgen
Today, we got workspaces running with loadgen, but, that left us with some questions and changes to make.

1. Fixes to support PVC on loadgen and integration test #12560

Integration tests
@jenting is starting to add basic support for PVC in our tests, I shared a comment as a guiding thought to help limit scope in the below issue.

1. [integration tests] add regular workspace PVC integration test from PVC test plans #12497

Questions

1. Do we still need chown for workspaces using PVC? May I ask you to track in a separate issue? This is something I think we can track and resolve in a separate issue but, isn't necessarily needed for dog fooding.
2. Is this issue still valid, or can we close now? [PVC] When using PVC some files in .git folder have root ownership instead of gitpod #12527