[ws-manager] gracefully shuts down workspace, leaving behind bound PVCs, avoiding backup of user data, after unknown event

[kylos101]

Bug description

In other words, we don't experience data loss, but, the pod stops gracefully, and when the user starts the workspace again, they would not have their data...even though we have it in a PV.

I tried deleting us72, but could not because there were two dangling PVC:

gitpod /workspace/gitpod (main) $ kubectl get pvc
NAME                                      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS             AGE
ws-ccd64f44-d3b6-49eb-9d1e-9275406745ef   Bound    pvc-35a16057-d21c-44e2-9a76-a36f44fb1866   30Gi       RWO            csi-gce-pd-g1-standard   47h
ws-eb6cb985-86f3-435b-9def-d820d2b9060a   Bound    pvc-50708c34-a721-4f3d-855e-f74c94e2c034   30Gi       RWO            csi-gce-pd-g1-standard   45h

For the first...given workspace logs and this workspace trace:

1. Startworkspace is logged on workspace
2. It cannot be scheduled (waiting for scale-up)
3. Start workspace is logged again after 7 minutes (still not scheduled to a node)
   1. Which lines up with us seeing startWorkspace error at 7m in the traces
   2. We poll for seven minutes, to see if the pending pod should be recreated, and startWorkspace called again
   3. We force delete the original pod and try starting again using original context 🤯
   • Introduced in Refactor Manager StartWorkspace #11547
4. csi provisioner is started for this workspace
5. Ring 0 stops, we must've landed on workspace-ws-us72-standard-pvc-pool-2dvw
6. workspace cannot connect to ws-daemon
7. workspace fails to start, the volume snapshot is empty
8. workspace fails to start is logged again

Steps to reproduce

This could be because of:

1. node scale-up that is too slow (given the logs, perhaps).
2. Another possibility, is that we restarted ws-manager "during a key moment", and it the snapshot did not proceed. We restarted ws-manager a couple times this week.
3. The 1h timeout, which is a byproduct from when we persisted using node storage and backed up using GCS

So, either:

1. Create an ephemeral cluster
2. Start many workspaces with loadgen, causing node scale-up, if it takes >7 minutes, you'll hit the code path that was involved for these two workspaces
3. Stop the workspaces
4. Check to see if they backed up, or, left behind PVCs.

or maybe

Stop a bunch of workspaces, and while they're stopping (before, during, and after snapshot) stop ws-manager.

or maybe

timeout after an hour

Workspace affected

gitpodio-templatetypesc-qxnleu3pzu4

Expected behavior

There are a few things:

1. We should try to backup for longer than 1 h, this way we do not have to manually snapshot PVCs before we delete workspace clusters.
2. We should have a metric track duration for when a PVC is bound but has no pod, and a trigger an alert when one or more exists for too long.

Questions:

1. The workspace affected was gracefully Stopped (not Failed or Stopping), which indicates the user could have tried restarting their workspace and not restored their files. This would have been very confusing because they will not have their uncommitted files restored. Is this expected?

Example repository

No response

Anything else?

We currently stop trying to backup after a 1h timeout. This was a design decision for object storage based backups, and should be revisited as part of PVC.

[sagor999]

As you mentioned, this could be related to several ws-man restarts in that cluster due to issues with it.

[kylos101]

Thanks @sagor999 ! I updated the description's steps to recreate to consider. I also updated the Anything Else to hint that the backup timeout is still 1h with PVC, and that we should reconsider that. For example, it could also explain why the PVCs were left dangling - I'm not sure if we log that the 1h timeout was hit. Can you check?

[kylos101]

@jenting can you focus on this issue? It is new (created on Friday), and seems like it'll impact our ability to upload user data in a timely manner, potentially causing an odd experience for workspace restarts (when snapshot has not been done yet).

[jenting]

A similar issue as #13282.

We force delete the original pod and try starting again using original context
In other words, we don't experience data loss, but, the pod stops gracefully, and when the user starts the workspace again, they would not have their data...even though we have it in a PV.

So, we recreate the start workspace request, and the 2nd workspace pod fails to start, correct?

[jenting]

Related instances log

• https://cloudlogging.app.goo.gl/SvVvU4ussCoUqqNJ9
• https://cloudlogging.app.goo.gl/goiLF5jT5hKdBPto6
• https://cloudlogging.app.goo.gl/5Majuy8uVsEgZGBv5

[kylos101]

So, we recreate the start workspace request, and the 2nd workspace pod fails to start, correct?

@jenting good observation, although, I see the same failures logged here too. Does a PVC already exist when we force delete the pod? Would it make sense to also force delete the PVC, and create a new one as part of the 2nd startWorkspace request?

[kylos101]

@jenting another consideration for this issue, is that the startWorkspace request could be for an existing workspace, rather than a new one. In the case of an existing workspace, what is the user experience like? For example, I assume the 2nd workspace never starts. But, if I try restarting the stopped workspace, is my data restored from the PVC snapshot?

[jenting]

Close this issue because it's duplicate as #13282