containerd newtask ip assignment failure

[mehrdadrad]

I'm trying to run a firecracker+container through this example w/ ip address:
https://github.com/firecracker-microvm/firecracker-containerd/blob/master/examples/taskworkflow.go
but i'm seeing this:

go run taskworkflow.go -ip 172.16.0.2 -gw 172.16.0.1 -mask 255.255.255.0
2019/03/05 00:30:39.642224 Creating containerd client
2019/03/05 00:30:39.642836 Created containerd client
2019/03/05 00:30:40.043966 Successfully pulled docker.io/library/nginx:latest image
2019/03/05 00:30:48.914751 creating task: type with url : not found: unknown
exit status 1

I created tap0 interface and assigned ip address .1 also it's up

tap0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.16.0.1  netmask 255.255.255.0  broadcast 0.0.0.0
        ether be:9b:37:dc:6e:f6  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

containerd logs:

time="2019-03-05T01:29:50.456796435Z" level=info msg="starting VM" runtime=aws.firecracker
time="2019-03-05T01:29:50.478100799Z" level=info msg="starting instance" runtime=aws.firecracker
time="2019-03-05T01:29:50.491380857Z" level=info msg="Attaching drive /dev/loop6, slot 0, root false."
time="2019-03-05T01:29:50.491843014Z" level=info msg="Attaching drive /var/lib/firecracker-containerd/runtime/hello-rootfs.ext4, slot root-drive, root true."
time="2019-03-05T01:29:50.513348515Z" level=info msg="calling agent" runtime=aws.firecracker
time="2019-03-05T01:29:50.513450168Z" level=warning msg="vsock dial failed (attempt 1 of 5), will retry in 100ms" error="no such device" runtime=aws.firecracker
time="2019-03-05T01:29:50.613639387Z" level=warning msg="vsock dial failed (attempt 2 of 5), will retry in 200ms" error="no such device" runtime=aws.firecracker
time="2019-03-05T01:29:50.814002433Z" level=warning msg="vsock dial failed (attempt 3 of 5), will retry in 400ms" error="connection reset by peer" runtime=aws.firecracker
time="2019-03-05T01:29:51.214380049Z" level=info msg="creating clients" runtime=aws.firecracker
time="2019-03-05T01:29:51.214434197Z" level=info msg="creating task 'demo21'" runtime=aws.firecracker
time="2019-03-05T01:29:51.492373233Z" level=error msg="create failed" error="rpc error: code = Unknown desc = type with url : not found" runtime=aws.firecracker

without ip address it works and the task creates successfully but w/ ip address it shows type with url : not found!!

firecracker version: 0.12.0
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
AWS i3.metal

please let me know if you need more information.

[aaithal]

Thanks for reporting this issue @mehrdadrad. I suspect that this error is coming from the runtime when it tries to unmarshal the protobuf from the client (taskworkflow):

firecracker-containerd/runtime/service.go

Line 606 in 0e39b73

cfg, err := typeurl.UnmarshalAny(opts)

Unfortunately, there isn't much being logged in this code path. Can you please check out this branch where I have added an error log for such a failure and rerun the example so that we can have some more logs around the failure?

I've created the #125 PR for the logging change as well.

[aaithal]

@aaithal I tried compiling your branch or master but the below error comes up:

make STATIC_AGENT='true'
make -C agent
make[1]: Entering directory '/root/go-contd/src/github.com/firecracker-microvm/firecracker-containerd/agent'
CGO_ENABLED=0 go build -installsuffix cgo -a -ldflags "-s" -o agent
# github.com/firecracker-microvm/firecracker-containerd/agent
./main.go:96:26: undefined: "github.com/containerd/containerd/runtime/v2/runc".New
Makefile:21: recipe for target 'agent' failed

That's super weird. Build seems to be passing in #125. Also, I just merged that PR to master. Can you please retry with the master branch now?

[mehrdadrad]

@aaithal i found the issue. i had to downgrade the containerd repo. it mentioned at docs

git checkout v1.2.1

[mehrdadrad]

@aaithal same error w/ new containerd-shim-aws-firecracker based on your branch.

[aaithal]

@aaithal same error w/ new containerd-shim-aws-firecracker based on your branch.

Does the additional log line show up now? I'm mostly looking for this one:

firecracker-containerd/runtime/service.go

Line 182 in bd82588

}).Error("failed to unmarshal task create request options")

[mehrdadrad]

@aaithal yes

go run taskworkflow.go -ip 172.16.0.2 -gw 172.16.0.1 -mask 255.255.255.0
2019/03/05 19:53:55.729859 Creating containerd client
2019/03/05 19:53:55.730640 Created containerd client
2019/03/05 19:53:56.028500 Successfully pulled docker.io/library/nginx:latest image
2019/03/05 19:54:04.110042 creating task: unmarshaling task create request options: unmarshaling task create request options: type with url : not found: unknown
exit status 1

[mehrdadrad]

I just added opts.Value to error and i can see the below:
string=
AA:FC:00:00:00:01tap0"lconsole=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw ip=172.16.0.2::172.16.0.1:255.255.255.0:::off::::):

[mehrdadrad]

it fixed, looks that's related to protobuf version!!