Unable to mount docker-socket with v0.4.0

[badeball]

Due to legacy reasons, we're still mounting the host's docker-socket into our containers. With v0.3.0 and by using setfacl on the socket, this worked fine. With v0.4.0 however, this seems to have ceased to work. I've put together an example using Vagrant to provision a VM that illustrates inability to call eg. docker info in such container.

There's nothing very particular going on in the Vagrantfile, prepare.sh or verify.sh. I pretty much just install sysbox, configure Docker and invoke setfacl with the IDs that sysbox has determined.

You can run the following to watch it fail on v0.4.0:

$ git clone https://gist.github.com/9ff3bd8caa2bdc0714e1274a54b01f36.git
$ cd 9ff3bd8caa2bdc0714e1274a54b01f36
$ vagrant up
...
    default: ++ docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock docker:latest docker info
    default: Client:
    default:  Context:    default
    default:  Debug Mode: false
    default: 
    default: Server:
    default: ERROR: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

.. and run the following to watch it succeed on v0.3.0:

$ git checkout master~
$ vagrant destroy -f
$ vagrant up
...
    default: ++ docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock docker:latest docker info
    default: WARNING: No swap limit support
    default: Client:
    default:  Context:    default
    default:  Debug Mode: false
    default: 
    default: Server:
    default:  Containers: 1
    default:   Running: 1
    default:   Paused: 0
    default:   Stopped: 0
    default:  Images: 1
    default:  Server Version: 20.10.7
    default:  Storage Driver: overlay2
    default:   Backing Filesystem: extfs
    default:   Supports d_type: true
    default:   Native Overlay Diff: true
    default:   userxattr: false
    default:  Logging Driver: json-file
    default:  Cgroup Driver: cgroupfs
    default:  Cgroup Version: 1
    default:  Plugins:
    default:   Volume: local
    default:   Network: bridge host ipvlan macvlan null overlay
    default:   Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
    default:  Swarm: inactive
    default:  Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc sysbox-runc
    default:  Default Runtime: sysbox-runc
    default:  Init Binary: docker-init
    default:  containerd version:
    default:  runc version:
    default:  init version:
    default:  Security Options:
    default:   apparmor
    default:   seccomp
    default:    Profile: default
    default:  Kernel Version: 5.4.0-88-generic
    default:  Operating System: Ubuntu 20.04.3 LTS
    default:  OSType: linux
    default:  Architecture: x86_64
    default:  CPUs: 2
    default:  Total Memory: 981MiB
    default:  Name: ubuntu-focal
    default:  ID: ZZ2B:ZM27:SGSN:7OVN:XDWS:5UJC:C6SI:Z6ZW:LECS:HOB4:KUHH:SINK
    default:  Docker Root Dir: /var/lib/docker
    default:  Debug Mode: false
    default:  Registry: https://index.docker.io/v1/
    default:  Labels:
    default:  Experimental: false
    default:  Insecure Registries:
    default:   127.0.0.0/8
    default:  Live Restore Enabled: false
    default: 

Run to clean up:

$ vagrant destroy -f
$ cd ..
$ rm -rf 9ff3bd8caa2bdc0714e1274a54b01f36

[badeball]

If you're unfamiliar with Vagrant, you can run vagrant ssh (in the directory where you ran vagrant up) to SSH into the VM, where you have sudo and can mess around.

[rodnymolina]

@badeball, thanks for filing this issue and for the reproduction details.

Problem is a direct consequence of the limitations of shiftfs when interacting with non-regular files (such as sockets). See that we're mounting shiftfs over the host resource that you are importing into the sys-container:

/ # cat /proc/self/mountinfo | egrep "docker.sock"
831 791 0:63 /docker.sock /run/docker.sock rw,relatime - shiftfs /var/lib/sysbox/shiftfs/4fe5ff48-9305-432c-a756-256d6f804ad3 rw

Starting in v0.4.0 we made some changes to the volume/bind-mounting logic to expand the scenarios in which shiftfs module is automatically utilized. This is the reason that you don't see this behavior in v0.3.0.

In essence, we rely on shiftfs when the host resources that the user is bind-mounting into the sys-container are not accesible due to a lack of permissions (i.e., mismatching uids/gids inside and outside the sys-container).

Now, in your setup, you are relying on setacl to grant access to the subordinate uid/gid of the sys-container, so technically there is no real need for shiftfs here. However, our current logic is not taking this pattern into account, so we'll need to adjust this.

As a workaround you can try to set this knob --bind-mount-id-shift=false in your sysbox-mgr's systemd file. By doing this you are telling Sysbox that you don't want shiftfs to be mounted over your host resources. That should work.

Hope that helps.

[ctalledo]

FYI, this will be fixed in the upcoming Sysbox v0.5.0 release by virtue of Sysbox supporting the Linux "ID-mapped mounts" feature (alternative to shiftfs that works on socket mounts). However, this requires a host with kernel >= 5.12.

[ctalledo]

Closing as this is fixed in the upstream code, and fix will be present in upcoming v0.5.0 release.