taskworkflow dies with snapshotter error

[ustiugov]

Dear Team,

I installed and launched firecracker-containerd following the quickstart guide. When I execute examples/taskworkflow.go, I face the following error:

2020/02/07 13:14:51.112672 Creating containerd client
2020/02/07 13:14:51.113477 Created containerd client
2020/02/07 13:14:52.203320 creating container: unpack: failed to stat snapshot sha256:531743b7098cb2aaf615641007a129173f63ed86ca32fe7b5a246a1c47286028: snapshotter not loaded: devmapper: invalid argument

At the same time, I don't see any containerd output when running taskworkflow. Manual commands from the quickstart guide work just fine.

Host: Ubuntu 18.04, kernel 4.15.0-76-generic

[ustiugov]

I've located the problem. In taskworkflow.go, the sock path is configured incorrectly:

-       containerdAddress      = "/run/containerd/containerd.sock"
+       containerdAddress      = "/run/firecracker-containerd/containerd.sock"

Also, taskworkflow.md is out of date a bit. For example, -mask argument has been removed.

Shall I submit a PR fixing those two bugs?

[ustiugov]

I tried to test taskworkflow with network configuration but there is the following problem:

ustiugov@iccluster027:~/staged-data-tiering/firecracker-containerd/examples$ sudo ./taskworkflow -ip 172.16.0.2 -gw 172.16.0.1
2020/02/07 16:55:04.203970 Creating containerd client
2020/02/07 16:55:04.204690 Created containerd client
2020/02/07 16:55:05.309439 failed to create VM: rpc error: code = Unknown desc = shim CreateVM returned error: rpc error: code = Unknown desc = failed to create VM: failed to build VM configuration: failed to convert network config StaticConfig:<MacAddress:"AA:FC:00:00:00:01" HostDevName:"tap0" IPConfig:<PrimaryAddr:"172.16.0.2" GatewayAddr:"172.16.0.1" > > : failed to parse CIDR from "172.16.0.2": invalid CIDR address: 172.16.0.2

Configuration:

sudo ip tuntap add tap0 mode tap
sudo ip addr add 172.16.0.1/24 dev tap0
sudo ip link set tap0 up
sudo iptables -t nat -A POSTROUTING -o ens786f0 -j MASQUERADE
sudo iptables -A FORWARD -i tap0 -o ens786f0 -j ACCEPT
ifconfig
tap0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.16.0.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether 06:76:14:0a:83:0c  txqueuelen 1000  (Ethernet)

[IRCody]

Shall I submit a PR fixing those two bugs?

Hey @ustiugov. We'd welcome a PR to fix those issues.

I'm not sure about the network configuration issue. @mxpv, @kzys , @xibz, any ideas?

[kzys]

Thanks for reporting the issue!

Yeah. It doesn't make much sense to use /run/containerd/containerd.sock while the docs are using /run/firecracker-containerd/containerd.sock. Our integration tests are running fine since it uses /run/containerd/containerd.sock inside Docker, but assuming customers already have containerd, using /run/firecracker-containerd/containerd.sock would be much easier since it wouldn't conflict with containerd.

[kzys]

Regarding the network configuration issue, after #257, taskworkflow.go assumes the -ip parameter to be a CIDR (172.16.0.2/24) rather than a single IP address (172.16.0.2) but we haven't updated the docs. Sorry for the confusion.

[ustiugov]

@kzys In order to update the taskworkflow #399 , one needs to change the tests... What would you suggest?

Yeah. It doesn't make much sense to use /run/containerd/containerd.sock while the docs are using /run/firecracker-containerd/containerd.sock. Our integration tests are running fine since it uses /run/containerd/containerd.sock inside Docker, but assuming customers already have containerd, using /run/firecracker-containerd/containerd.sock would be much easier since it wouldn't conflict with containerd.

[kzys]

I personally think it would be better to use /run/firecracker-containerd/containerd.sock everywhere.

[ustiugov]

since #399 was merged, this issue can be closed