[ws-manager] Stuck in stopping: cannot initialize workspace

[geropl]

Instead of hanging in "stopping" forever these workspaces should become "stopped" eventually.

Event trace log: https://cloudlogging.app.goo.gl/QJY327RqEMDBPbrNA

DB query:

SELECT wsi.*
	FROM d_b_workspace_instance AS wsi
    JOIN d_b_workspace AS ws
		ON ws.id = wsi.workspaceId
    WHERE wsi.phasePersisted = 'stopping'
        AND status->>'$.conditions.failed' LIKE "cannot initialize workspace%"
	ORDER BY wsi.creationTime DESC
    LIMIT 500;

instanceIds:

'50bff6fd-2b1f-4d33-8b74-5362739add6f'
'9f422515-e048-4a38-80b0-1b3c90e2775e'
'909c1ffc-ec77-4933-a2b1-b56f57ddb09b'
'a9dc5638-06d6-47ce-89ee-ec724d8adb73'
'b159250b-8aa0-4bc7-8fa2-dc677f61ba54'
'bdd55e4a-5804-44ef-ba99-1613fbb66d24'
'fc7de8c8-1154-427b-8bb5-0432149aa678'
'9a3faa95-8050-4578-a680-b1b87d35c356'
'6d127543-f026-40c0-9caa-bceb7d21e822'
'a997a894-a986-48bc-93f1-d8a626327150'
'6cb0356c-2e65-4355-8830-12e7f86f86e4'
'45bd88d6-e7cf-4d2a-97bb-cd147dfb363f'

[svenefftinge]

We should look into the cause of this failures. There might be different reasons, but the logs reveal that there seem to be many more cases than those that stuck in stopping in DB.

[JanKoehnlein]

/schedule

for further investigation

[csweichel]

/assign