Support optionally forwarding agent and systemd journal logs from within VM to the host for debugging

[sipsma]

It's often very helpful for debugging to have logs from the agent process running inside the VM. Getting those logs is not straightforward as by default they will stay inside the VM.

One quick hack to get them is this change, which forwards the agent logs over /dev/console to the runtime process on the host, which in turn forwards them to containerd logs.

That approach can't be merged into master though as it results in a lot of noise in the containerd logs and could possibly lead to heavy use of /dev/console, which has been known to result in performance issues in the past. A mergable solution to this would need to deal with those issues; some ideas to explore are only forwarding them in a debug mode and making the destination of the logs configurable (could be containerd logs, could be a file, etc.).

[kzys]

Yes. It is unsettling to see some of the logs go nowhere. Regarding the console, firecracker-microvm/firecracker#792 also mentions that the console is slow.

I think it may make sense to collect all of systemd logs. Issues like #253 cannot be seen by just looking the agent.

[sipsma]

Agree about users having the ability to collect all systemd logs.

I think it's worth seeing if we can find a way for systemd journal logs to get streamed out of the VM over vsock, which thanks to the new Firecracker vsock implementation, could mean that logs are available on the host at a unix socket, which could be convenient for users.

I don't know enough about systemd, but if there is a way to write a journald plugin or similar that forwards logs over an AF_VSOCK socket, that could be a decent generic solution to this problem.

[mmonaco]

Hey, we're streaming structured (json) journald over vsock. It works well enough but can be flaky. I came here looking for best practices / prior art on what others are doing.