[self-hosted] Workspace will not start with error: cannot mount proc

[jgallucci32]

Bug description

When starting a workspace it hangs at the "pulling container image..." screen and fails to start with the error message "cannot mount proc" reported by both the ws-daemon and the workspace pod.

ws-daemon

{
  "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
  "error": "cannot open tree:
    github.com/gitpod-io/gitpod/ws-daemon/pkg/iws.moveMount
        github.com/gitpod-io/gitpod/ws-daemon/pkg/iws/iws.go:512
  - function not implemented",
  "level": "error",
  "message": "cannot mount proc",
  "procPID": 29095,
  "reqPID": 27,
  "serviceContext": {
    "service": "ws-daemon",
    "version": ""
  },
  "severity": "ERROR",
  "time": "2021-09-10T04:42:05Z"
}

workspace

{
  "@type": "type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent",
  "error": "rpc error: code = Internal desc = cannot mount proc",
  "level": "error",
  "message": "cannot mount proc",
  "ring": 1,
  "serviceContext": {
    "service": "workspacekit",
    "version": ""
  },
  "severity": "ERROR",
  "time": "2021-09-10T04:42:05Z"
}

Steps to reproduce

1. Deploy Gitpod on self-hosted k8s cluster
2. Launch new workspace

Expected behavior

Workspace successfully launches

Example repository

n/a

Anything else?

Gitpod v0.10.0
RKE 1.20.10 (using cri-containerd and not dockershim)
containerd 1.4.9 (installed via Docker CE 20.10.7)
Red Hat Enterprise Linux 8.3

NOTE: Docker CE is installed, but the RKE kubelet is configured to use containerd, bypassing Docker. This gets past the ring0 problem described in #3909

[jgallucci32]

This might be caused by the same issue with runc 1.0.1 fixed in PR #5139.

[jgallucci32]

Still getting the same error message with 0.10.0-nightly. The iws.go module is still throwing the error "cannot mount proc". The PR mentioned above only fixes the notification from the workspacekit, but ws-daemon is still throwing an error in this case.

[jgallucci32]

The offending line of code is here

gitpod/components/ws-daemon/nsinsider/main.go

Line 286 in f496a74

fd, _, errno := unix.Syscall(unix.SYS_OPEN_TREE, uintptr(dfd), uintptr(unsafe.Pointer(p1)), flags)

Per the Linux manuals, the syscall open_tree() required Linux kernel version 5.2 or greater. This means CentOS and older Ubuntu distros will be unsupported for Gitpod.
https://man7.org/linux/man-pages/man2/syscalls.2.html

[aledbf]

This means CentOS and older Ubuntu distros will be unsupported for Gitpod.

Yes, this is documented https://www.gitpod.io/docs/self-hosted/latest/requirements#minimum-kubernetes-requirements
Also, keep in mind components as agent-smith uses eBPF (one of the reasons for the kernel version)

[jgallucci32]

@aledbf Thanks for the info. Are there others besides open_tree() and agent-smith driving the newer kernel requirements?

[aledbf]

two things are user namespace support for FUSE > v4.18 and OverlayFS > v3.18

@csweichel ping for additional info

[ghuntley]

two things are user namespace support for FUSE > v4.18 and OverlayFS > v3.18

Added to https://github.com/gitpod-io/website/pull/955

[aledbf]

@ghuntley not sure that is necessary. Requiring a kernel > v5.4 is enough.

[jgallucci32]

After upgrading from RHEL 8.3 to Ubuntu 20.04 (with no other changes to components) the workspaces now launch successfully. This confirms later version of Gitpod will not run on RHEL 8 and you must have at least Kernel 5.2 to operate.

/close

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.