Prebuild process gets stuck (when updating `.gitpod.Dockerfile`)

[shaal]

Bug description

When updating .gitpod.Dockerfile and pushing it back to repo on a branch, the prebuild seems stuck.
It takes over half an hour (but I am not sure how long, perhaps 1 hour?) where all I see is this screen with jumping G.
(while console doesn't display any error)

Coming back to the computer after a while, I see this screen -

And here's the output I found in console -
https://gist.github.com/shaal/b34b1e2ffba2d50deb4e42a3b4ed961c

I checked gitpod.io/workspaces, and I see 5(?) workspaces running, even that I only opened 2 windows of the following URL, expecting to see the prebuild output - https://gitpod.io/#https://github.com/shaal/DrupalPod/tree/debug-with-tmate

Steps to reproduce

In my experience, it is difficult to replicate or anticipate how prebuild behave (especially with custom docker image build)
This happened to me also the other day, when I worked on https://github.com/phase2/outline/tree/gitpod, when I opened an issue in https://community.gitpod.io/t/prebuild-is-stuck/4171

I think I usually use the manual prebuild using the #prebuild/ in the URL, but in repos that has the Gitpod bot, I would get a message that a prebuild is already running.

These workspaces supposed to run prebuild, but they are stuck, and once 4 workspaces are "running", I cannot open another one, but at the same time I cannot stop a prebuild that is stuck like that.
So, I usually wait a few hours, then the workspaces are shutdown, and I can run prebuild using the #prebuild/ in the URL, and then it finally works.

Expected behavior

No response

Example repository

No response

Anything else?

No response

[shaal]

Update 1:

One of the workspaces (https://gitpod.io/start/#black-ferret-dd1kh8y8) that seemed to be stuck on "Build an image" screen and empty output, finally has some (error output).
Console says: /start: started workspace instance: d8a47bd4-07f0-4cfe-8ac9-3a43dac47463
Screenshot:

The actual output in 'terminal preview'

Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                 Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                  Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                                   Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                                                    Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
               Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                                 Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                                                                                      Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                 Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".
                                                                                      Error: 13 INTERNAL: cannot resolve base image ref: Error response from daemon: manifest unknown: Failed to fetch "e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455" from request "/v2/gitpod-dev/base-images/manifests/e737d8012fc85bce44573b7673879707ba3aa94a2dc313806e6a3e3edea8f455".

[shaal]

Update 2:

Gitpod seems to keep restarting these prebuilds, you can see below that these are 4 new workpsaces.
In the meantime, I cannot do anything. I cannot stop these workspaces from rebuilding, and I cannot work on other things in the meantime (because it's maxed out on 4 active workspaces)

[shaal]

Update 3:

After a few hours, it was no longer stuck.
I opened a workspace with the updated repo, and I saw the "build image" page, now displaying terminal messages as I expected. (I was able to SSH into the process of image build, since I use tmate in there)

[shaal]

Update 4:

I added ENV TRIGGER_REBUILD 1 to .gitpod.Dockerfile, and now I see this screen, which seems to be stuck again. No console errors, and nothing displayed in the "terminal" output.

After 1-2 hours:

[ghuntley]

https://gitpod.io/#https://github.com/ghuntley/learn-graphql/tree/gh/gitpodify

Confirming that I've also witnessed the following as of moments ago

now I see this screen, which seems to be stuck again. No console errors, and nothing displayed in the "terminal" output.

[geropl]

Outch... this feels like a restart-cycle because we miss some signal. Might have been introduced with the recent headless-log changes. 😕 Will dig into this shortly.

[geropl]

Thx so much @shaal for this treasure trove of data.

It's a bit entangled, bit it seems like here are multiple strange things (bugs?) happening here:

• the root cause for the cannot resolve base image ref: not sure where this comes from, yet
• we seem to not terminate prebuilds properly which do not reach the kubernetes control plane (to be verified!)
• we start new workspaces in a loop on a (failed) prebuild

Will try to verify and fix one-by-one.

[shaal]

Thank you for the quick response, let me know if I can help testing.

[geropl]

Ok, pinned it down:

• the frontend does not properly end it's re-try loop to load headless logs but assumes we will navigate away once we have the first clear signal on "prebuild done". If there is an error during workspace start before we generate an ideUrl (e.g., during imagebuilds) we can never navigate away from the prebuild logs view.
• we do not handle the image builder error properly: instead of failing and stopping the workspace it runs into a timeout (1h) it seems the image build takes very long (longer than the default timeout of 1h), which, in combination with a race condition in image-builder and the frontend trying to start a lot of workspaces at once, results in these error messages.

Besides the first error which we're going to fix in #4875 there are two issues left:

• UX: Dead end when image-build fails during prebuild #4879: one is a UI/UX gap for exactly this situation (no ideUrl but workspace start fails) which this PR will not attempt to fix because it needs a bit more discussion. I will open a new issue for that.
• the other is the race condition in image-builder and (maybe) potential to improve the reporting of build errors. But as we're planning to migrate to a new service very soon I bet we won't do anything about that for now.