[PVC] restart the ws-manager, the ws-manager sometimes panic because the concurrent map read and map write

[jenting]

Bug description

If there are lots of VolumeSnapshot is ready, restart the ws-manager sometimes panic because the concurrent map read and map write.

gitpod/components/ws-manager/pkg/manager/monitor.go

Lines 206 to 209 in 6604264

	if m.notifyPod[podName] == nil {
	m.notifyPod[podName] = make(chan string)
	}
	m.notifyPod[podName] <- vsc

goroutine 224 [running]:
runtime.throw({0x1c9bd02?, 0x48?})
        runtime/panic.go:992 +0x71 fp=0xc0008d3920 sp=0xc0008d38f0 pc=0x438411
runtime.mapaccess1_faststr(0xc00051c700?, 0x4?, {0xc000642c30, 0x2d})
        runtime/map_faststr.go:22 +0x3a5 fp=0xc0008d3988 sp=0xc0008d3920 pc=0x412de5
github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Monitor).onVolumesnapshotEvent(0xc00092e990, {{0x1c70877?, 0x8?}, {0x1f37a60?, 0xc00015b680?}})        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/monitor.go:206 +0x60d fp=0xc0008d3e78 sp=0xc0008d3988 pc=0x179eb0d
github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Monitor).handleEvent(0xc00092e990, {{0x1c70877?, 0x0?}, {0x1f37a60?, 0xc00015b680?}})
        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/monitor.go:160 +0x4f fp=0xc0008d3eb0 sp=0xc0008d3e78 pc=0x179e44f
github.com/gitpod-io/gitpod/ws-manager/pkg/manager.(*Monitor).handleEvent-fm({{0x1c70877?, 0xc0004ce280?}, {0x1f37a60?, 0xc00015b680?}})
        <autogenerated>:1 +0x45 fp=0xc0008d3ee8 sp=0xc0008d3eb0 pc=0x17b2ce5
github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool.(*EventWorkerPool).work(0xc000405e50, 0x0?)
        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool/workpool.go:124 +0x26a fp=0xc0008d3f90 sp=0xc0008d3ee8 pc=0x1779d6a
github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool.(*EventWorkerPool).workUntilStopped(0xc000405e50, 0x0?)
        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool/workpool.go:99 +0x45 fp=0xc0008d3fc0 sp=0xc0008d3f90 pc=0x1779aa5
github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool.(*EventWorkerPool).Start.func1()
        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool/workpool.go:76 +0x2a fp=0xc0008d3fe0 sp=0xc0008d3fc0 pc=0x177978a
runtime.goexit()
        runtime/asm_amd64.s:1571 +0x1 fp=0xc0008d3fe8 sp=0xc0008d3fe0 pc=0x46b181
created by github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool.(*EventWorkerPool).Start
        github.com/gitpod-io/gitpod/ws-manager/pkg/manager/internal/workpool/workpool.go:76 +0x34

Steps to reproduce

Prepare lots of VolumeSnapshot and is ready to use, restart the ws-manager, sometimes it panic because the concurrent map read and map write.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

#7901

[kylos101]

@jenting it looks like this is an issue for PVC, could I ask you to add it to the related epic?