jailed VM creation fails on Amazon Linux 2 #345
Description
firecracker-containerd currently uses jail locations under /run. On AL2 (and presumably CentOS and other RHEL variants) this filesystem is mounted with the nodev flag, so the device nodes created by runc are not usable.
firecracker-containerd debug logs show:
INFO[2019-11-26T22:15:06.264606168Z] Successfully ran jailer handler jailer=runc ociBundlePath=/run/firecracker-containerd/firecracker-containerd-example/fc-example runcBinaryPath=/usr/local/bin/runc runtime=aws.firecracker vmID=fc-example
INFO[2019-11-26T22:15:06.264668138Z] Called startVMM(), setting up a VMM on /run/firecracker-containerd/firecracker-containerd-example/fc-example/rootfs/api.socket runtime=aws.firecracker vmID=fc-example
WARN[2019-11-26T22:15:06.381824962Z] firecracker exited: exit status 1 runtime=aws.firecracker vmID=fc-example
WARN[2019-11-26T22:15:06.381911059Z] Failed handler "fcinit.StartVMM": Firecracker did not create API socket /run/firecracker-containerd/firecracker-containerd-example/fc-example/rootfs/api.socket: 1 error occurred:
* exit status 1
runtime=aws.firecracker vmID=fc-example
ERRO[2019-11-26T22:15:06.381939624Z] error="failed to create VM: failed to start the VM: Firecracker did not create API socket /run/firecracker-containerd/firecracker-containerd-example/fc-example/rootfs/api.socket: 1 error occurred:\n\t* exit status 1\n\n" runtime=aws.firecracker vmID=fc-example
ERRO[2019-11-26T22:15:06.382144578Z] error="shim CreateVM returned error: rpc error: code = Unknown desc = failed to create VM: failed to start the VM: Firecracker did not create API socket /run/firecracker-containerd/firecracker-containerd-example/fc-example/rootfs/api.socket: 1 error occurred:\n\t* exit status 1\n\n"
DEBU[2019-11-26T22:15:06.383322943Z] shim has been terminated error="signal: killed" vmID=fc-example
...which is not the most helpful error.
The location used for VM jailing should be configurable. Appropriate locations will vary from site to site.