Various prebuild issues

[shaal]

Bug description

I've been working a lot with prebuilds, and some issues are still not resolved:

1. When creating a PR and running a prebuild for the FIRST time, you can go to the Gitpod's URL of the project (ie: https://gitpod.io/#https://github.com/drud/ddev) and you can see all the commands that the prebuild is running.
   But when making new commits after that, and you'll only see "Connecting to workspace logs...".
   A prebuild IS RUNNING in the background, but there's no visual representation of what's happening, and when prebuild is done the workspace is reloaded on that browser tab.
   Related: Project Configuration - Run Prebuild always shows "Connecting to workspace Logs..." #6305
   

2. I set a special webhook to send me an email when a prebuild is running (https://github.com/shaal/DrupalPod/blob/ready-made-envs/.gitpod.yml#L8), every time a prebuild is running, I am getting 2 emails, so for some reason the prebuild is called and running twice, but I don't know why.

3. (Noticeable on projects with init start-tasks that take a long time, and perhaps related to the previous issue of prebuild running twice) -
   After manually initiating a prebuild (or pushing a commit), if you go to the Gitpod URL of the project, you will see the Prebuild page (which is expected After Prebuild finish running), and the Gitpod workspace opens, instead the new workspace using the prebuild that just finished, it runs all the init tasks of the prebuild all over again.
   If you close this workspace, and open a new workspace, now prebuild is trully reloaded, and only command tasks are running in the workspace.

Steps to reproduce

1. Fork repo with long init tasks (ie. https://github.com/shaal/DrupalPod)
2. Create a commit and push it.
3. Watch the prebuild process (issues 2 and 3) + workspace that opens right after.
4. Create another commit and push it.
5. Watch the prebuild process (issues 1 and 2 and 3) + workspace that opens right after.

Workspace affected

No response

Expected behavior

1. Prebuild should only run once per commit (or manual trigger).
2. Workspace should never run prebuild tasks in the workspace.
3. Prebuild should always display what commands are running while prebuild is running.

Example repository

No response

Anything else?

No response

[AlexTugarev]

@shaal, thanks for reporting!

We're looking into issues with flaky logs. /cc. @geropl

Let me try to reproduce the other two issue...

[JanKoehnlein]

/schedule

[roboquat]

@JanKoehnlein: Issue scheduled in the meta team (WIP: 0)

In response to this:

/schedule

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[AlexTugarev]

Quick update: I'm still trying to reproduce with a fork of the DrupalPod repo. Currently trying to create a prebuild for ready-made-envs, which failed for unknown reasons.

The error reads headless task failed: context canceled, which would be a timeout?! The last log lines are

Package operations: 28 installs, 0 updates, 0 removals
  - Installing dflydev/dot-access-data (v1.1.0): Extracting archive
  - Installing consolidation/output-formatters (4.1.2): Extracting archive
  - Installing consolidation/annotated-command (4.4.0): Extracting archive
  - Installing consolidation/log (2.0.2): Extracting archive
  - Installing consolidation/self-update (2.0.0): Extracting archive
  - Installing doctrine/inflector (1.4.4): Extracting archive
  - Installing doctrine/event-manager (1.1.1): Extracting archive
  - Installing doctrine/collections (1.6.8): Extracting archive
  - Installing doctrine/cache (1.12.1): Extracting archive
  - Installing doctrine/persistence (1.3.8): Extracting archive
  - Installing drupal/admin_toolbar (3.0.3): Extracting archive
  - Installing doctrine/common (2.13.3): Extracting archive
  - Installing drupal/devel (4.1.1): Extracting archive
  - Installing webmozart/path-util (2.3.0): Extracting archive
  - Installing webflo/drupal-finder (1.2.2): Extracting archive
  - Installing nikic/php-parser (v4.13.0): Extracting archive
  - Installing psy/psysh (v0.10.9): Extracting archive
  - Installing league/container (3.4.1): Extracting archive
  - Installing grasmash/yaml-expander (1.4.0): Extracting archive
  - Installing enlightn/security-checker (v1.9.0): Extracting archive
  - Installing grasmash/expander (1.0.0): Extracting archive
  - Installing consolidation/config (1.2.1): Extracting archive
  - Installing consolidation/site-alias (3.1.1): Extracting archive
  - Installing consolidation/site-process (4.1.0): Extracting archive
  - Installing consolidation/robo (3.0.6): Extracting archive
  - Installing consolidation/filter-via-dot-access-data (1.0.0): Extracting archive
  - Installing chi-teck/drupal-code-generator (1.33.1): Extracting archive
  - Installing drush/drush (10.6.1): Extracting archive
7 

Given that, if that's the situation here, then it would be logical to see the init tasks being executed on start up, just because there is no successful prebuild tarred & uploaded.

On the Connecting to workspace logs... issue with the logs: While we still have the issue with short prebuilds and @geropl already has an idea what's wrong in there, I couldn't see that issue with more complex projects like this. But there is one more thing. Prebuilds are scheduled differently to regular workspaces and startup times might vary. Unfortunately there is no differentiation of that in the awaiting state.

[shaal]

@AlexTugarev if you have a few minutes, we can coordinate a quick call on Gitpod Discord, I'll be happy to walk you through replicating each one of the issues I wrote about.

[shaal]

I noticed that you can see prebuilds are running twice in the UI (on same commit, 2 prebuilds running):
I pushed a commit to a branch, which triggered the 2 simultaneous prebuilds -