Wrong branch checked out (main) when opening a Pull Request

[jankeromnes]

Bug description

When opening a Pull Request, we expect Gitpod to check out the Pull Request's branch.

However, since recently, it seems that it (sometimes?) checks out the default branch instead. 🐛

Steps to reproduce

1. https://gitpod.io/#https://github.com/gitpod-io/gitpod/pull/13963
2. Checked out branch should be jx/fix-attribution-again, but you actually get main instead:

Workspace affected

gitpodio-gitpod-m35tlw96nof, gitpodio-gitpod-4nxderxz5x0

Expected behavior

1. Gitpod should always check out the Pull Request branch

Example repository

https://github.com/gitpod-io/gitpod

Anything else?

No response

[laushinka]

This also happens when opening a non-PR branch[1].

[jankeromnes]

In the database, the affected workspaces appear to have the correct context:

[{
  "id": "gitpodio-gitpod-m35tlw96nof",
  "creationTime": "2022-10-19T09:44:12.466Z",
  "ownerId": "643ac637-74ae-4f82-9a86-c8527eb1e496",
  "contextURL": "https://github.com/gitpod-io/gitpod/pull/13963",
  "description": "[server] Don\u0027t attribute usage to a team without active billing",
  "context": "{\"title\":\"[server] Don\u0027t attribute usage to a team without active billing\",\"repository\":{\"cloneUrl\":\"https://github.com/gitpod-io/gitpod.git\",\"host\":\"github.com\",\"name\":\"gitpod\",\"owner\":\"gitpod-io\",\"private\":false},\"ref\":\"jx/fix-attribution-again\",\"refType\":\"branch\",\"revision\":\"fdc1d3b969fbe186310435832b149dcc45b6798a\",\"nr\":13963,\"base\":{\"repository\":{\"cloneUrl\":\"https://github.com/gitpod-io/gitpod.git\",\"host\":\"github.com\",\"name\":\"gitpod\",\"owner\":\"gitpod-io\",\"private\":false},\"ref\":\"main\",\"refType\":\"branch\"},\"normalizedContextURL\":\"https://github.com/gitpod-io/gitpod/pull/13963\",\"checkoutLocation\":\"gitpod\",\"snapshotBucketId\":\"0915f180-d85c-4832-9533-a2f13e4af2eb\",\"prebuildWorkspaceId\":\"f06a627a-0d4a-4b5c-9f81-7f7b992900b6\",\"wasPrebuilt\":true}",
  "config": "{\"image\":\"eu.gcr.io/gitpod-core-dev/dev/dev-environment:af-dev-update-image.8\",\"workspaceLocation\":\"gitpod/gitpod-ws.code-workspace\",\"checkoutLocation\":\"gitpod\",\"ports\":[{\"port\":1337,\"onOpen\":\"open-preview\"},{\"port\":3000,\"onOpen\":\"ignore\"},{\"port\":3001,\"onOpen\":\"ignore\"},{\"port\":3306,\"onOpen\":\"ignore\"},{\"port\":4000,\"onOpen\":\"ignore\"},{\"port\":5900,\"onOpen\":\"ignore\"},{\"port\":6080,\"onOpen\":\"ignore\"},{\"port\":7777,\"onOpen\":\"ignore\"},{\"port\":9229,\"onOpen\":\"ignore\"},{\"port\":9999,\"onOpen\":\"ignore\"},{\"port\":13001,\"onOpen\":\"ignore\"},{\"port\":13444},{\"port\":8022,\"onOpen\":\"ignore\"}],\"tasks\":[{\"name\":\"Install Preview Environment kube-context\",\"command\":\"previewctl install-context --watch\\nexit\\n\"},{\"name\":\"Add Harvester kubeconfig\",\"command\":\"./dev/preview/util/download-and-merge-harvester-kubeconfig.sh\\nexit 0\\n\"},{\"name\":\"Installer dependencies\",\"init\":\"(cd install/installer \u0026\u0026 make deps)\\nexit 0\\n\"},{\"name\":\"Java\",\"command\":\"if [ -z \\\"$RUN_GRADLE_TASK\\\" ]; then\\n  read -r -p \\\"Press enter to continue Java gradle task\\\"\\nfi\\nleeway exec --package components/supervisor-api/java:lib --package components/gitpod-protocol/java:lib -- ./gradlew build\\nleeway exec --package components/ide/jetbrains/backend-plugin:plugin-latest --package components/ide/jetbrains/gateway-plugin:publish-latest --parallel -- ./gradlew buildPlugin\\n\"},{\"name\":\"TypeScript\",\"before\":\"scripts/branch-namespace.sh\",\"init\":\"yarn --network-timeout 100000 \u0026\u0026 yarn build\"},{\"name\":\"Go\",\"before\":\"pre-commit install --install-hooks\",\"init\":\"./components/gitpod-protocol/go/scripts/generate-config.sh\\nleeway exec --filter-type go -v -- go mod verify\\n\",\"openMode\":\"split-right\"}],\"vscode\":{\"extensions\":[\"EditorConfig.EditorConfig\",\"golang.go\",\"hashicorp.terraform\",\"ms-azuretools.vscode-docker\",\"ms-kubernetes-tools.vscode-kubernetes-tools\",\"stkb.rewrap\",\"zxh404.vscode-proto3\",\"matthewpi.caddyfile-support\",\"heptio.jsonnet\",\"timonwong.shellcheck\",\"fwcd.kotlin\",\"dbaeumer.vscode-eslint\",\"esbenp.prettier-vscode\"]},\"jetbrains\":{\"goland\":{\"prebuilds\":{\"version\":\"stable\"}}},\"_origin\":\"repo\",\"_featureFlags\":[]}",
  "archived": "0",
  "shareable": "0",
  "imageSource": "{\"baseImageResolved\":\"eu.gcr.io/gitpod-core-dev/dev/dev-environment:af-dev-update-image.8\"}",
  "imageNameResolved": "eu.gcr.io/gitpod-dev/workspace-images:7097d5fde6cd08f1131fdc232f920696501d86d6f6a83d5d8774590aeb787baf",
  "_lastModified": "2022-10-19 09:44:12.474817 UTC",
  "deleted": "0",
  "type": "regular",
  "baseImageNameResolved": "eu.gcr.io/gitpod-core-dev/dev/dev-environment@sha256:3dc66ca8903c3270befbbf6e38b2fcee4fd7bd6e0ff71aaa83e79f526f6a64dd",
  "softDeleted": null,
  "pinned": "0",
  "softDeletedTime": "",
  "contentDeletedTime": "",
  "basedOnPrebuildId": "f06a627a-0d4a-4b5c-9f81-7f7b992900b6",
  "basedOnSnapshotId": "",
  "projectId": "ef9d3f23-e5ab-4086-82f4-91813524bc3e",
  "cloneURL": "https://github.com/gitpod-io/gitpod.git"
}]

EDIT: Here is the same parsed "context" as above but in more readable format:

{
  "title": "[server] Don't attribute usage to a team without active billing",
  "repository": {
    "cloneUrl": "https://github.com/gitpod-io/gitpod.git",
    "host": "github.com",
    "name": "gitpod",
    "owner": "gitpod-io",
    "private": false
  },
  "ref": "jx/fix-attribution-again",
  "refType": "branch",
  "revision": "fdc1d3b969fbe186310435832b149dcc45b6798a",
  "nr": 13963,
  "base": {
    "repository": {
      "cloneUrl": "https://github.com/gitpod-io/gitpod.git",
      "host": "github.com",
      "name": "gitpod",
      "owner": "gitpod-io",
      "private": false
    },
    "ref": "main",
    "refType": "branch"
  },
  "normalizedContextURL": "https://github.com/gitpod-io/gitpod/pull/13963",
  "checkoutLocation": "gitpod",
  "snapshotBucketId": "0915f180-d85c-4832-9533-a2f13e4af2eb",
  "prebuildWorkspaceId": "f06a627a-0d4a-4b5c-9f81-7f7b992900b6",
  "wasPrebuilt": true
}

[jankeromnes]

We no longer believe the revert fixed this.

[jankeromnes]

Here is a different way to reproduce the bug:

1. https://gitpod.io/#https://github.com/gitpod-io/gitpod/blob/4c59bdae5db15a9e439c1264771dac55f67a0db4/README.md
2. I would expect commit 4c59bdae5db15a9e439c1264771dac55f67a0db4 to be checked out (see context URL), but instead, it is the older commit 1a8f1b6e5ac55697a505a75aefe385021030ca84 that is checked out

[jankeromnes]

Useful debug info for #13991 (comment):

• The created workspace gitpodio-gitpod-6khsn6o5ku0 has:
  • context.revision: 4c59bdae5db15a9e439c1264771dac55f67a0db4
  • basedOnPrebuildId: 43656625-7109-4c8a-9f34-10a21ce7b289
• The (incremental) prebuild 43656625-7109-4c8a-9f34-10a21ce7b289 has:
  • commit: 4c59bdae5db15a9e439c1264771dac55f67a0db4
  • buildWorkspaceId: gitpodio-gitpod-w71vpiz4807
• The buildWorspace gitpodio-gitpod-w71vpiz4807 has:
  • context.revision: 4c59bdae5db15a9e439c1264771dac55f67a0db4
  • basedOnPrebuildId: 631842a0-5214-4cf8-a271-ecb0d03a3baa (the base prebuild)
• The (base) prebuild 631842a0-5214-4cf8-a271-ecb0d03a3baa has:
  • commit: 1a8f1b6e5ac55697a505a75aefe385021030ca84

It looks like the very old commit that initially served to build the base prebuild never got reset to a more recent commit (not when building the more recent incremental prebuild, nor when opening the workspace for a more recent commit).

[geropl]

Two more comments:

• note that we see that the Contexts stored in the DB are correct: so we seem to be passing the right config to the initializer
• we ruled out that the two recent changes to the workspace creation code caused this.

Given this, we suspect it's either a) a change in content-initializer, or b) caused by the PVC enabled prebuilds.

/vv @kylos101