Prebuilds are stuck in 'queued'

[svenefftinge]

Bug description

The DB contains +6k entries of prebuilds in 'queued' state. Would be good to understand what happens, clean things up and prevent these from being created further.

Updating prebuilt.state on failed prebuild start

During this incident we discovered that there seems to be an ongoing issue with updating state for prebuilds whose instances never reach RUNNING phase (note the empty startedTime):

SELECT wsi.id, wsi.creationTime, wsi.phasePersisted, wsi.status, wsi.startedTime, wsi.stoppingTime, wsi.stoppedTime
   FROM d_b_workspace_instance AS wsi
   JOIN d_b_prebuilt_workspace AS pws
      ON pws.buildWorkspaceId = wsi.workspaceId
   WHERE pws.state = 'queued'
	AND wsi.phasePersisted = 'stopped'
	ORDER BY wsi.creationTime DESC;

Note: Especially older entries could also be fallout from other incidents + cleanups, but there seems to be a couple of fresh ones every day.

Steps to reproduce

select * from d_b_prebuilt_workspace where state = 'queued' order by creationTime;

[svenefftinge]

The cases have in common that the phase is unknown (which has a timeout of 600 secs vs. 3600 secs for preparing).
The code here sets the DB entries to stopping and stopped after that time.

Unfortunately, I couldn't find any signs in the log of workspace-cluster of these workspace instances. Also weren't lucky with finding traces in honeycomb (I checked 20 instances) 😞 .

[geropl]

The cases have in common that the phase is unknown (which has a timeout of 600 secs vs. 3600 secs for preparing).

This sounds related to a fix @andrew-farries is working on here. At some point we seem to have started using "unknown" as the initial phase - which does not make sense to me, at all. 😕 We're seeking to fix this now.

@andrew-farries Can you make sure we introduce separate timeouts for "building", and start out with the same value as "preparing"?

[svenefftinge]

Yes, that explains why they are no signs in workspace-cluster the workspaces are probably building docker images and timing out in the meantime (after 10min).

[jldec]

Suspected example of affected user (internal)
Is there a workaround or DB change we can use to mitigate?

[geropl]

Having had a look at the data we indeed leak a couple of prebuilds in queued every day.
There seem to be two constellations:

1. image build timed out (after 1h): we currently have not (reliable) way to ensure that whenever an image build fails, we also fail the prebuild, because we assume we're watching the imagebuild from server from start -> end. IMO this can only be solved either by:
   1. making the imagebuild 💯 visible inside webapp, by storing image-build workspaces just like regular and prebuilds 👎
   2. making imagebuild completely transparent to webapp, by encapsulating them inside workspace 👍
2. prebuild workspaces are stopped after 10mins no image builds, and no idea why those are stopped and by whom (yet) by MetaInstanceController: We currently have a timeout of 10mins for imagebuilds, which is a left over from before the "preparing" | "building" phase split We have a timeout for 1h for imagebuilds (correct), but the "building" phase just got deployed two days ago, so that's why we still see recent entries in the DB. Will:
   1. increase the timeout for "building" to 1h (so it's aligned with the imagebuild timeout itself): this is temporary, and expected to become obsolete once we solved 1. above
   2. make sure that if we stop a workspace, we also check for and update the corresponding prebuild if necessary