[PVC] Prebuild without PVC, user A account with PVC and trigger rebuild, user B account without PVC

[jenting]

Bug description

The prebuild project without PVC.

The user A account with PVC, add new branch X and commit, push branch X to the repo, trigger prebuild on Gitpod. The prebuild runs without PVC.
The user B account without PVC, open that branch X, the user B uses PVC.

Steps to reproduce

The prebuild projects without PVC.

• The user A account with PVC.
• The user B account without PVC.
• User A adds new branch X and commit, pushes branch X to the repo, and triggers prebuild on Gitpod. The prebuild runs without PVC. 👍
• User B opens branch X, the user B uses PVC (The prebuild project and the user's account are both without PVC).

Workspace affected

No response

Expected behavior

User B should open the workspace without PVC because the prebuild project is configured without PVC, also the user's account does not enable PVC.

Example repository

No response

Anything else?

#7901

[kylos101]

👋 @jenting I just tested this, and got same result. Here are supporting details:

project has pvc disabled, https://gitpod.io/t/prebuild-experiment/prebuild-experiment

user has PVC enabled, https://gitpod.io/admin/users/8df3495b-685d-46e0-9820-009cc3b4afd8

push a code change to a branch

prebuild will not use PVC, which is expected, because it's disabled on the project. Details:

node: headless-ws-us64-internal-xl-pool-nhjr
workspace: kylos101-prebuildexperi-5p1e6ngpkxf
pod: prebuild-880d87db-2e38-45e1-8ada-5bfa79f94210

Same user has PVC disabled now

PVC is still disabled on the project

A new workspace is started from the branch's prebuild, but has PVC. Details:

node: workspace-ws-us64-large-pvc-pool-kfwz
workspace: kylos101-prebuildexperi-biberrbv8vk
pod: ws-f3d2657c-4e47-4ee1-aca3-c557e6bcc9e3 

I think there are two contributing factors:

1. We're sending the persistent_volume_claim feature flag when triggering prebuild
2. We're persisting the persistent_volume_claim feature flag to the prebuild workspace

This query shows the feature flag is being persisted:

SELECT JSON_EXTRACT(dbw.config, "$._featureFlags") 
FROM gitpod.d_b_workspace dbw 
where dbw.id = 'kylos101-prebuildexperi-5p1e6ngpkxf';

If we avoid sending the persistent_volume_claim feature flag on the startWorkspace request for prebuild (unless it is enabled on the project), would that help? Or do you think it would be better to avoid persisting that particular feature flag to the DB and continue sending? Or something else? @sagor999 for 👀

Either way, this is a good catch, nice find! 💪

[jenting]

I think when open the workspace, we should check the prebuild project PVC enabled or the user's PVC enabled.

If one of the condition hits, we send start workspace request with feature flag persistent_volume_claim. Otherwise, the start workspace request must not contains feature flag persistent_volume_claim.

[kylos101]

👋 @jenting ,

I think when we open the workspace, we should check the prebuild project PVC enabled or the user's PVC enabled.

A couple questions:

1. Do you mean prebuild workspace or regular workspace?
2. Why check prebuild project PVC setting, or user PVC feature flag?

Considerations:

1. the PVC setting on the project (we check this now, it controls if the prebuild runs with PVC or not)
2. the PVC feature flag on the user (we check this now, it controls if new regular workspaces start with PVC or not)
3. the feature flag persisted to the prebuild workspace config field in the database (if this is set, we should start regular workspaces with PVC, I'd argue the problem is we're wrongly setting this...especially given the scenario I shared ☝️, and to fix, maybe we have to avoid persisting it in the first place when the project has PVC disabled )
4. the feature flag persisted to the regular workspace config field in the database (this should control whether a regular workspace restart uses PVC or not 🤞 )

It seems like the problem is, when we run prebuilds, if the startWorkspace request contains persistent_volume_claim, and the project has PVC disabled, we're persisting persistent_volume_claim to the workspace's config JSON object in the database. So, as a result, when we start regular workspaces from the prebuild, they use PVC (because the config object says it was used).

[jenting]

1. Do you mean prebuild workspace or regular workspace?

regular workspace

2. Why check prebuild project PVC setting, or user PVC feature flag?

Because if the project was built with PVC, then for any user who open the regular workspace from that prebuild will force to use PVC.

Because the user enables PVC, open workspace from prebuild (with PVC or without PVC) or from git clone, the regular workspace should use PVC.

---

From my point of view, we should save the persistent_volume_claim feature flag in DB (d_b_workspace?) if the prebuild workspace was built with the PVC. Otherwise, we do not save persistent_volume_claim feature flag in DB (d_b_workspace?).

Then, the start workspace request persistent_volumn_claim feature flag determine by

• the prebuild workspace with PVC, add it.
• the prebuild workspace without PVC:
  • the user account with PVC enabled, add it.
  • the user account without PVC enabled, do not add it.

[kylos101]

I like your idea, @jenting . My only concern with user feature flag in its current form, is that it is hard to roll out. For example, we have to set it via Admin page, or SQL query.

Ideally we'd set the persistent_volume_claim feature flag for regular workspace start from the server component using rules that we define in configcat. This way its not exclusively at the user level.

Checkout #12745 for the idea, should help us rollout in a more flexible way.