Remove directories for removed containers on the host

[samuelkarp]

[ec2-user@ip-172-31-30-18 ~]$ sudo /usr/local/bin/ctr run --snapshotter firecracker-naive --runtime aws.firecracker --tty docker.io/library/busybox:latest test
/ # echo hello
hello
/ # exit
[ec2-user@ip-172-31-30-18 ~]$ sudo /usr/local/bin/ctr c rm test
[ec2-user@ip-172-31-30-18 ~]$ sudo /usr/local/bin/ctr run --snapshotter firecracker-naive --runtime aws.firecracker --tty docker.io/library/busybox:latest test
ctr: mkdir /run/containerd/io.containerd.runtime.v2.task/default/test: file exists: unknown

Even though I've deleted the container (and its snapshot) on the host, the microVM rootfs the host retains the bundle directory that was created for the container. We should either clean up the bundle directory when the container is removed (or stopped?), or we should ensure that the microVM rootfs is read-only (or separate per microVM).

[samuelkarp]

Per discussion with @IRCody yesterday, he thinks that the error relates to a path on the host instead of inside the microVM. When I look on the host, I do see what he's talking about:

[ec2-user@ip-172-31-30-18 ~]$ sudo ls /run/containerd/io.containerd.runtime.v2.task/default/test
address  config.json  firecracker.sock	log  rootfs  shim.pid  work

I'm not sure what is supposed to clean up this directory, maybe the runtime?

[IRCody]

I think the runtime is supposed to clean it up. I think this comment is wrong and we actually do need to clean some stuff up here.

Reading the docs it looks like there are two scenarios:

• containerd will call cleanup() over RPC.
• containerd is expecting to call the runtime binary with the "delete" subcommand when RPC calls fail expects that runtime binary to cleanup any state left by the other one. That delete call gets processed here which calls cleanup.

[samuelkarp]

There appear to be additional leftover files in /run/containerd/fifo.

[samuelkarp]

I was able to fix the non-deletion locally, but that seems to have exposed another bug where subsequent containers launched with the same ID seem to have the IO streams disconnected after launch (i.e., the firecracker and runtime processes remain running but ctr exits with an error).