CI skips test_drives.py::test_rescan_dev

[alexandruag]

Looking at logs for various PRs, it seems our CI skips test_rescan_dev from integration_tests/functional/test_drives.py on x86_64 for no apparent reason. The test is not skipped when running on a local dev machine.

The relevant part in the CI log is something like:

integration_tests/functional/test_drives.py resize2fs 1.44.1 (24-Mar-2018)
Resizing the filesystem on /tmp/firecracker_test_session/17f55cd2-26ee-4ebb-be10-a2cae034e07e/fsfiles/scratch.ext4 to 524288 (1k) blocks.
The filesystem on /tmp/firecracker_test_session/17f55cd2-26ee-4ebb-be10-a2cae034e07e/fsfiles/scratch.ext4 is now 524288 (1k) blocks long.

.s....

[andreeaflorescu]

This is skipped when you cannot create a loopback device on the host. Actual code here:

firecracker/tests/integration_tests/functional/test_drives.py

Line 100 in abfa7a5

except CalledProcessError as error:

PR that introduced the test: #1371

[kzys]

Yes. losetup somehow didn't work on the CI host.

https://firecracker-pr.s3.amazonaws.com/pr-logs/01371/kzys-firecracker-2d8dd18.log

losetup: /dev/loop8: failed to set up loop device: No such file or directory

and @aghecenco recommended to comment out the test for a moment and log an issue on GitHub. I did the former but forgot the latter.

[alxiord]

The CI AMIs have the kernel module loaded but the first losetup fails.
Detailed failure:

2018: loopdev:      CXT: [0x7fffb3e44280]: initialize context
2018: loopdev:      CXT: [0x7fffb3e44280]: init: ignore ioctls
2018: loopdev:      CXT: [0x7fffb3e44280]: init: loop-control detected 
2018: loopdev:      CXT: [0x7fffb3e44280]: find_unused requested
2018: loopdev:      CXT: [0x7fffb3e44280]: using loop-control
2018: loopdev:      CXT: [0x7fffb3e44280]: loop0 name assigned
2018: loopdev:      CXT: [0x7fffb3e44280]: find_unused by loop-control [rc=0]
2018: loopdev:      CXT: [0x7fffb3e44280]: set backing file=/tmp/firecracker_test_session/c917d717-c69c-4cc9-ad05-a08f87ea8
2018: loopdev:    SETUP: [0x7fffb3e44280]: device setup requested
2018: loopdev:    SETUP: [0x7fffb3e44280]: backing file open: OK
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: open /dev/loop0 [rw]: No such file or directory
2018: loopdev:    SETUP: [0x7fffb3e44280]: failed [rc=-2]
losetup: /tmp/firecracker_test_session/c917d717-c69c-4cc9-ad05-a08f87ea82b1/fsfiles/fs2.ext4: failed to set up loop device: No such file or directory
2018: loopdev:      CXT: [0x7fffb3e44280]: de-initialize
2018: loopdev:     ITER: [0x7fffb3e44438]: de-initialize

The losetup --find ... command issued by the integration test should create a loop device if none are found, but doesn't appear to do so. A losetup invocation before running the tests should fix it.

[alxiord]

The /dev/loop* devices aren't created by losetup - at least initially. They can be manually created with mknod if the loop module is loaded.

[alxiord]

@firecracker-microvm/compute-capsule

This can be fixed by mknod'ing any missing /dev/loop* devices, but I don't like it as it will affect the state of the host outside the container. WDYT?

[dhrgit]

We seem to have hit a mix of a few issues that lead to losetup not working correctly inside our dev container:

• the host kernel doesn't have the loop module loaded;
• consequently, /dev/loopX block devices aren't present at container start time;
• the container /dev space is set only once, at container start;
• losetup causes the loop module to be loaded, after container started, so /dev/loopX show up on the host, but not inside the container; therefore, losetup fails inside the container;
• a possible workaround could be to bind-mount /dev; however, this doesn't work because we also launch the container with a default init process (docker run --init), and older versions of docker used /dev/init as the fixed location for its init process (tini)

There are different ways to approach this, but the one I'm inclined towards is to rework our container to use tini from a location of our choosing (e.g. /sbin/tini), meaning we no longer need to rely on the docker --init switch and the conflicting /dev/init.

[alxiord]

There are different ways to approach this, but the one I'm inclined towards is to rework our container to use tini from a location of our choosing (e.g. /sbin/tini), meaning we no longer need to rely on the docker --init switch and the conflicting /dev/init.

I'll give that a shot 🔫