Allow users to "force stop" workspaces which are stuck in stopping

[kylos101]

Is your feature request related to a problem? Please describe

This generally happens if the pod backing the related workspace enters a bad state while stopping.

Describe the behaviour you'd like

After some preconditions (an hour being in stopping?), grant the user the ability to force stop their workspace through the dashboard.

Why? This gives the user the ability to access their workspace and files w/o needing manual intervention from Gitpodders.

If this feature is used, we should track it via some metric, so we can understand how often it is being used.

Also, we should consider displaying a warning to the user, prior to using. For example, it is possible their backup is not done. So, if they were to force stop a workspace, and start a new instance, it could load a backup with N-1 data.

Another interesting idea is option #3 (below). It's less destructive, and the user would also be able to add those files to a new workspace, w/o potentially disrupting a long running backup.

Describe alternatives you've considered

1. Rely on personnel to manually fix such conditions via the database. This is not ideal because it is error prone and creates a significant delay between the user and them accessing their files for the workspace. Plus, some users may not even be reaching out for support, and just lose their data.
2. Automatically force stop any regular workspace which are stuck in stopping after some precondition.
3. Give the user the ability to download their files, w/o having the ability to force stop a workspace.
4. Add a Force Stop button to the admin page, and have personnel use this button (rather than a database query)

Additional context

#7597 (comment)

Front conversations

[kylos101]

@geropl bumping this due to recent events last week. We lost a few nodes over the course of a few days, and roughly 60 workspaces were stuck in stopping.

[kylos101]

Context thread on Discord.

[jimmybrancaccio]

This would have been helpful to have had when working on (cnv_jisdoqy) in Front.

[jimmybrancaccio]

Another instance when this would have been helpful to have (cnv_jiw5gju).

[jimmybrancaccio]

This would have been helpful to have had in cnv_jkhffwq.

[cstickel]

Actually we've a workspace stuck in stopping since friday. So one more incident where this would have helped, if that boosts the priority. What's the best way to get it unstuck for now?

[alessandrosp]

+1. Having the same problem.

[axonasif]

Hi @alessandrosp, you might be experiencing issues because of an incident today.
Please check https://www.gitpodstatus.com/

[easyCZ]

After some preconditions (an hour being in stopping?), grant the user the ability to force stop their workspace through the dashboard.

Can we offer this as a regular feature, without the constraint on a workspace being in stopping for some time? It feels that as a user, I should own the workspace, and hence even the force stop. We of-course need to communicate what it means to force-stop and the implications, but it feels that this should be a first-class feature rather than a special case of the workspace being stuck in Stopping (because if it is, it feels like a hack where we're pushing that problem onto the user).