Sometimes workspace attempts to start with PVC feature enabled

[sagor999]

Bug description

Sometimes workspace attempts to start with PVC feature enabled, even though user does not have flag enabled, nor the project has flag enabled.
StartWorkspace request example:

{
    "id": "fdec6ee9-73a1-4104-9490-a9de771db03f",
    "metadata": {
        "metaId": "matiasmillan-bc9-mcx26mbrw4b",
        "owner": "b2761e91-264e-454c-b811-f8b54367e4f2",
        "project": "a681734e-f4c4-46b2-89e0-de801d31262d"
    },
    "servicePrefix": "matiasmillan-bc9-mcx26mbrw4b",
    "spec": {
        "class": "default",
        "deprecatedIdeImage": "eu.gcr.io/gitpod-core-dev/build/ide/code:commit-079f161f0aaf95d6b5e28f0709b3e629337291e9",
        "envvars": [
            {
                "name": "GITPOD_IDE_ALIAS",
                "value": "intellij"
            },
            {
                "name": "GITPOD_WORKSPACE_CONTEXT_URL",
                "value": "<redacted>"
            },
            {
                "name": "GITPOD_WORKSPACE_CONTEXT",
                "value": "{"isFile":false,"path":"","title":"<redacted>","ref":"main","refType":"branch","revision":"<redacted>","repository":{"cloneUrl":"<redacted>,"host":"github.com","name":"<redacted>","owner":"<redacted>","private":false,"fork":{"parent":{"cloneUrl":"<redacted>","host":"github.com","name":"<redacted>","owner":"<redacted>","private":false}}},"normalizedContextURL":"<redacted>","checkoutLocation":"<redacted>","upstreamRemoteURI":"<redacted>","prebuildWorkspaceId":"9c8960b7-0839-4a29-aaf8-ee0d72fafb1e","wasPrebuilt":true}"
            },
            {
                "name": "GITPOD_TASKS",
                "value": "[{"init":"./gradlew build"}]"
            },
            {
                "name": "VSX_REGISTRY_URL",
                "value": "https://open-vsx.gitpod.io/"
            },
            {
                "name": "SUPERVISOR_DOTFILE_REPO"
            },
            {
                "name": "[redacted]",
                "value": "[redacted]"
            }
        ],
        "featureFlags": [
            "PERSISTENT_VOLUME_CLAIM"
        ],
        "git": {
            "email": "<redacted>",
            "username": "<redacted>"
        },
        "ideImage": {
            "desktopRef": "eu.gcr.io/gitpod-core-dev/build/ide/intellij:commit-86cafcf14fd1b24b39d7b692f943f15cd5d7a04a",
            "supervisorRef": "eu.gcr.io/gitpod-core-dev/build/supervisor:commit-46cca98b280151eabccae382ddfbf42d8f2e2f2c",
            "webRef": "eu.gcr.io/gitpod-core-dev/build/ide/code:commit-079f161f0aaf95d6b5e28f0709b3e629337291e9"
        },
        "initializer": {
            "prebuild": {
                "git": [
                    {
                        "checkoutLocation": "<redacted>",
                        "cloneTaget": "main",
                        "config": {
                            "authPassword": "[redacted]",
                            "authUser": "oauth2",
                            "authentication": "BASIC_AUTH"
                        },
                        "remoteUri": "<redacted>",
                        "targetMode": "REMOTE_BRANCH",
                        "upstreamRemoteUri": "<redacted>"
                    }
                ],
                "prebuild": {
                    "fromVolumeSnapshot": true
                }
            }
        },
        "timeout": "30m",
        "volumeSnapshot": {
            "volumeSnapshotHandle": "projects/workspace-clusters/global/snapshots/snapshot-08cedb1c-bd7e-4610-a50c-4494d1c47db9",
            "volumeSnapshotName": "00b5a1d5-ca93-444d-b57d-046afa10935f"
        },
        "workspaceImage": "<redacted>",
        "workspaceLocation": "bc9"
    }
}

As you can see in this request, it attempts to restore workspace from volume snapshot, but that volume snapshot belongs to completely different workspace: gitpodio-templatepython-y5xa16s143e

I suspect that something is not right somewhere in this code:

gitpod/components/server/src/workspace/workspace-starter.ts

Lines 1361 to 1372 in b97e21a

	let volumeSnapshotId = lastValidWorkspaceInstanceId;
	// if this is snapshot or prebuild context, then try to find volume snapshot id in it
	if (SnapshotContext.is(workspace.context) || WithPrebuild.is(workspace.context)) {
	volumeSnapshotId = workspace.context.snapshotBucketId;
	}
	let volumeSnapshotInfo = new VolumeSnapshotInfo();
	const volumeSnapshots = await this.workspaceDb.trace(traceCtx).findVolumeSnapshotById(volumeSnapshotId);
	if (volumeSnapshots !== undefined) {
	volumeSnapshotInfo.setVolumeSnapshotName(volumeSnapshots.id);
	volumeSnapshotInfo.setVolumeSnapshotHandle(volumeSnapshots.volumeHandle);
	}

Steps to reproduce

n\a
happens randomly rarely

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

[sagor999]

Going to have to wait for #11824 to get deployed, to be able to continue investigation.

[sagor999]

Getting odd results in log of server:
https://cloudlogging.app.goo.gl/ABiP6R4qVxdEMmi78

It does not log this entry at all:

gitpod/components/server/src/workspace/workspace-starter.ts

Line 1372 in 6390f20

volSnapshotId: volumeSnapshotId,

@gitpod-io/engineering-webapp any idea why that field is not getting logged? (Not very familiar with TypeScript, maybe it does some magic behind the back).
And I need to get that entry logged to understand where snapshot id is coming from. Those workspaces should not be using volume snapshots (PVC), yet somehow they attempt to.

[geropl]

any idea why that field is not getting logged?

@sagor999 Jep: That method takes arguments in order: LogContext, message, error, payload, where precedence order for parsing is (roughly): message, LogContext, payload, error`

The following would work as expected:

log.info(
                "starting workspace with volume snapshot info",
                {
                    lastInstanceId: lastValidWorkspaceInstanceId,
                    ...
                },
);