Failed to download OTS in US cluster (possibly happens for prebuilds, only)

[csweichel]

Bug description

Looking at the logs we're seeing a surprising amount of OTS download failures as part of workspace content initialisation: https://console.cloud.google.com/logs/query;query=%22cannot%20download%20OTS%22%0A;timeRange=PT24H;cursorTimestamp=2022-02-08T14:43:32Z?project=workspace-clusters

Each of those failures is likely to yield a failed workspace - at least if the repo was private.
Possible contributing factors:

(edited by Sven)

• as this seems to happen only in prebuilds (only US cluster and lots of this error messages in d_b_prebuild_workspace), it could be that for some reason the time between the OTS is created and when it gets requested is longer than 30min (the lifetime of a token). Prebuild clusters are sometimes heavily packed so maybe there is just too much time in scaling up etc.
• we attempt to download the OTS multiple times for some reason. That's most likely a bug in the initializer. Checking the server logs and/or adding metrics would help identifying this.

As part of a fix, we should introduce OTS download failure metrics and keep an eye on them.

Steps to reproduce

Check the logs

[kylos101]

Also interesting.

[sagor999]

gitpod/components/content-service/pkg/initializer/initializer.go

Line 303 in a8e15a6

for i := 0; i < otsDownloadAttempts; i++ {

According to this line we will attempt to download OTS at most 10 times, with 1 second sleep in between each attempt.
This PR adds metrics to track this #8148

As for load balancer question, that is probably for webapp team? @gitpod-io/engineering-webapp

[geropl]

As for load balancer question, that is probably for webapp team?

I think we should focus on unifying our DBs into one, so we don't need to expose app cluster identity to users. Especially given how long this has been in prod, and how resource constraint we as a team are (/cc @JanKoehnlein ).

As a bandaid, we could extend this timeout to 20s, which should in all but very rare cases plenty of time to include the db-sync roundtrip.