[self-hosted] Workspace initialisation fails with error on PrepareForUserNS with moby

[pavan-tri]

Bug description

When the workspace starts, the ws-daemon/workspace-kit is trying to initialize ring0 and makes a GRPC call PrepareForUserNS. This call times out, leading it to stopping of intialization. WS go into "stopped" state.

nodeRoot was changed to "moby" as per troubleshooting guide for Self-hosted.

First Observation: Containerd wrapper is having a hardcoded namespace ( k8s.io Link) . When I changed the nodeRoot to moby, the underlying containerd runtime is expecting a "moby" name space. This is leading to no Container ID found error messages

Second Observation: Containerd wrapper also expects c.Labels to have map entries like "io.kubernetes.pod.name". With moby namespace the only map entry available is "com.docker/engine.bundle.path"

Steps to reproduce

Env Details:

AWS - EKS 1.16
Ubuntu 18.04 AMI
Containerd: 1.3.3-0ubuntu1~18.04.4

ws-daemon config:

workspaceSizeLimit: "5g"
nodeRoots:

• /var/lib
• /run/containerd/io.containerd.runtime.v1.linux/moby

Expected behavior

WorkSpace should start.

Example repository

No response

[ghuntley]

Howdy @pavan-tri can you please try v0.10.0 and report back? We have put together an EKS guide over at https://www.gitpod.io/docs/self-hosted/latest/installation

[pavan-tri]

Thanks for the update. I will keep you posted.

[stefanstoeckigt]

I get the same issue with v0.10.0, been stuck on this for a while.

[jgallucci32]

I am currently seeing the same issue with v0.10.0 on RHEL 8 with Docker CE 20.10.7. I understand Docker is not officially supported, but it has been working for over a year with 0.6.0 on both RHEL7 and RHEL8 hosts.

[jgallucci32]

I found a workaround for the first issue (hard-coded "k8s.io" namespace). You can change the default with the docker daemon as follows

cat << EOF > /etc/docker/daemon.json
{
  "containerd-namespace": "k8s.io"
}
EOF

Reboot the host for changes to take effect. This will change the default namespace for all containers to be "k8s.io". This also allows you to use the default nodeRoot in the Helm chart.

UPDATE: These instructions are OUT OF DATE. Will not work with Docker daemon. See updated comment.

[jgallucci32]

Regarding the second issue, I see labels for io.kubernetes.container.name and io.kubernetes.pod.name on the running container, but I do NOT seem them for io.cri-containerd.kind which is the third constant in that section of the code for labels.

This seems to be less a containerd issue and more a CRI plugin issue for containerd since containerd-cri is required for Gitpod per #2691

[jgallucci32]

I was able to get containerd-cri enabled using an installation of Docker CE. To enable it you must do the following:

1. Enable CRI plugin by modifying /etc/containerd/config.toml as follows

disabled_plugins = []

2. Run systemctl restart containerd
3. Verify cri plugin is enabled by running ctr plugins ls | grep cri
4. Modify kubelet to use containerd instead of dockershim (this example is for RKE cluster.yaml)

    kubelet:
      extra_args:
        container-runtime: remote
        container-runtime-endpoint: 'unix:///run/containerd/containerd.sock'
      extra_binds:
        - '/var/gitpod:/var/gitpod'
        - '/var/lib/containerd:/var/lib/containerd'

NOTE: The /var/gitpod binding is required for RKE deployments because kubelet is containerized

I used these two sources as reference coming up with the configuration
https://gianarb.it/blog/kubernetes-1-20-dockershim-in-practice
https://kinvolk.io/docs/flatcar-container-linux/latest/container-runtimes/switching-from-docker-to-containerd-for-kubernetes/