Epic: Run k3s natively in Gitpod

[fntlnz]

Right now, to run k3s in Gitpod the only viable option is to use emulation to create a VM as showed here https://github.com/gitpod-io/template-k3s

I did some analysis of why, even if we are able to run Docker, k3s still does not work.

There are a couple of errors to solve

Error 1 Snapshotter (:heavy_check_mark: )

ERRO[2021-07-21T08:57:21.159608731Z] Failed to configure agent: "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native": failed to mount overlay: invalid argument

Solution:

This is already solved, we can use --snapshotter=fuse-overlayfs since we fixed fuse support in #4594 and #4762

Error 2 Privileges (:negative_squared_cross_mark: )

E0721 09:00:38.387724   20938 server.go:288] "Failed to run kubelet" err="failed to run Kubelet: failed to create kubelet: open /dev/kmsg: no such file or directory"

The kubelet can run in rootless mode to avoid us dealing with privileged devices, like we do for Docker.

However, running in that mode

 sudo k3s server --snapshotter=fuse-overlayfs --rootless
WARN[2021-07-21T09:01:57.964650205Z] cannot set cgroup2 evacuation, make sure to run k3s as a systemd unit 
WARN[2021-07-21T09:01:57.965501805Z] Running RootlessKit as the root user is unsupported. 
WARN[2021-07-21T09:01:57.965778310Z] The host root filesystem is mounted as "". Setting child propagation to "" is not supported. 
FATA[2021-07-21T09:01:57.969073870Z] failed to setup UID/GID map: failed to compute uid/gid map: No subuid ranges found for user 0 ("root") 

Amazing, we can create the uid and gid maps for that user by editing /etc/subuid and /etc/subgid

However, after doing that

sudo k3s server --snapshotter=fuse-overlayfs --rootless
WARN[2021-07-21T09:09:54.117789745Z] cannot set cgroup2 evacuation, make sure to run k3s as a systemd unit 
WARN[2021-07-21T09:09:54.119755834Z] Running RootlessKit as the root user is unsupported. 
WARN[2021-07-21T09:09:54.120024215Z] The host root filesystem is mounted as "". Setting child propagation to "" is not supported. 
FATA[2021-07-21T09:09:54.125484467Z] failed to setup UID/GID map: newuidmap 24262 [0 0 1 1 200000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
: exit status 1 

Looks like the problem is that the current workspace root user cannot write the uid map of any process.

24548 openat(AT_FDCWD, "/proc/24262/", O_RDONLY|O_DIRECTORY) = 3
24548 openat(3, "uid_map", O_WRONLY)    = -1 EACCES (Permission denied)
24548 write(2, "newuidmap: open of uid_map faile"..., 53) = 53

Front conversations

[csweichel]

/schedule

[fntlnz]

I think we have two roads for this.

@leodido suggested that we can make the process dumpable (better than making dumpable system wide echo 1 > /proc/sys/fs/suid_dumpable

prctl(PR_SET_DUMPABLE, 1, 0, 0, 0)

Not sure about the security implications of this just yet

OR

Maybe it's a crazy and wrong idea but what if we use our seccomp notify mechanism to pass an unprivileged file handle and return that instead of the actual one in the proc filesystem?

[csweichel]

Maybe it's a crazy and wrong idea but what if we use our seccomp notify mechanism to pass an unprivileged file handle and return that instead of the actual one in the proc filesystem?

Probably a stupid question, but would that require SECCOMP_NOTIFY_IOCTL_ADDFD?

[leodido]

I think we have two roads for this.

@leodido suggested that we can make the process dumpable (better than making dumpable system wide echo 1 > /proc/sys/fs/suid_dumpable

prctl(PR_SET_DUMPABLE, 1, 0, 0, 0)

Not sure about the security implications of this just yet

OR

Maybe it's a crazy and wrong idea but what if we use our seccomp notify mechanism to pass an unprivileged file handle and return that instead of the actual one in the proc filesystem?

Yup! This code in the Kernel could be the reason why that write fails.

First attempt would be to play with the sysctl knob (/proc/sys/fs/suid_dumpable at the moment is set to 2).

[leodido]

A quick update on the progress made by me and @fntlnz (also thanks to @csweichel 🤗 ).

^^^ These are the settings for /etc/subuid and /etc/subgid we used.

Then inspecting with strace:

[pid 16099] openat(AT_FDCWD, "/dev/kmsg", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 16099] openat(AT_FDCWD, "/etc/localtime", O_RDONLY) = 7
[pid 16099] write(2, "\33[31mFATA\33[0m[2021-07-21T16:00:0"..., 92FATA[2021-07-21T16:00:01.195076930Z] no such file or directory                    
) = 92

Creating /dev/kmsg (in the host) made us progress further.

The next error is about permissions on /root/.rancher/.... Specifying a directory (or playing with symlinks) in /workspace removes such an error.

So, we got k3s to start in rootless mode. But it's not properly functioning yet.

For example:

gitpod /workspace/gitpod $ sudo k3s kubectl get pods --all-namespaces
Unable to connect to the server: x509: certificate signed by unknown authority

[csweichel]

/assign @fntlnz

[csweichel]

/assign @leodido