Prebuilds failing with "failed OTS download"

[geropl]

Context: #8096

We see a lot of workspaces (prebuilds only?) failing to retrieve the Git one-time-secret (OTS) when trying to run the initializer. Affected workspaces can be identified in the DB with:

SELECT *
FROM d_b_prebuild_workspace AS pws
WHERE pws.error LIKE "%failed OTS download%"
LIMIT 50;

It's possible that there are more affected, who's state is later overwritten by #8096

[geropl]

This issues is superseeded by two other issues:

• workspaces in general have trouble to either a) download the OTS or b) use the materialized key (both symptoms of the same root cause, tracked in Accessing server API at gitpod.io from within workspaces might fail sometimes #8703 )
• sometimes prebuilds hang for too long (<30mins) in the workspace cluster, resulting in start failures like these ☝️ While this is bad, both workspace and webapp teams have that our their radars, so this does not warrant a new issues.