100% cpu pinning processes starting kdevtmpfsi on host somehow

[chrisdlangton]

i've isolated the symptom to this container by shutting this off for 2 days and no issues until i cycled through each container 1 by 1 and kdevtmpfsi started on my host soon after this container ran.

After killing the VM host that was clearly infected and starting a new VM host to run docker, i replicated the behavior, there is definitely a security hole in linuxserver/nextcloud docekr containr.

Also during my investigation i noticed user abc starting php-fpm7 nginx processes - is this expected? strange user name choice if it is typical behavior.

This is my docker service call;

docker create \
  --name=nextcloud \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Australia/Melbourne \
  -p 3002:80 \
  -v $HOME/nextcloud/config:/config \
  -v $HOME/nextcloud/data:/data \
  --restart unless-stopped \
  linuxserver/nextcloud:17.0.1-ls52@sha256:5b928942971bca3ce30e012c8fa8694c41180b7f6e91804d5d271ace7c9085a6

That was the latest version yesterday (locking to latest and using sha256 is self-defeating when latest bumps and breaks the hash verification).

I'm going to try to generate a seccomp for this container, i'll add here if successful.

it might also be worth removing the SUID flag inside the containers when you are building them, i typically have luck using;

RUN for i in find / -path /proc -prune -o -perm /6000 -type f; do chmod a-s $i; done

This "may" prevent some container breakouts.

EDIT: it is worth mentioning the SUID flag is only problematic in a few circumstances, the most obvious being when a user inside docker matches one of the host (like root which you're using). An better solution is declaring a unique USER in the Dockerfile

[chrisdlangton]

tried my best all day, it was way too much for my usual strace use so i went to sysdig for this one, came up with the following profile;

{
    "defaultAction": "SCMP_ACT_ERRNO",
    "architectures": [
        "SCMP_ARCH_X86_64",
        "SCMP_ARCH_X86",
        "SCMP_ARCH_X32"
    ],
    "syscalls": [
        {
            "name": "",
            "action": "SCMP_ACT_ALLOW",
            "args": [],
            "names": [
                "setpgid",
                "capset",
                "mmap",
                "fstat",
                "recvfrom",
                "lseek",
                "umask",
                "times",
                "fstatfs",
                "splice",
                "clone",
                "rename",
                "signalfd",
                "socketpair",
                "epoll_create1",
                "ioctl",
                "setgroups",
                "execve",
                "setgid",
                "getdents64",
                "sched_getaffinity",
                "fork",
                "poll",
                "openat",
                "writev",
                "bind",
                "getpid",
                "brk",
                "socket",
                "rt_sigprocmask",
                "fchown",
                "chdir",
                "mknod",
                "utimensat",
                "capget",
                "accept",
                "setitimer",
                "getgid",
                "readv",
                "shutdown",
                "vfork",
                "getegid",
                "kill",
                "set_tid_address",
                "faccessat",
                "fcntl",
                "link",
                "io_setup",
                "mremap",
                "close",
                "connect",
                "rt_sigsuspend",
                "statfs",
                "geteuid",
                "pwritev",
                "exit_group",
                "getsockopt",
                "madvise",
                "fsync",
                "lstat",
                "newfstatat",
                "sysinfo",
                "chmod",
                "getppid",
                "fadvise64",
                "setsockopt",
                "chown",
                "clock_gettime",
                "access",
                "wait4",
                "nanosleep",
                "recvmsg",
                "exit",
                "gettid",
                "rmdir",
                "write",
                "setuid",
                "fchmod",
                "ftruncate",
                "prctl",
                "listen",
                "fchownat",
                "getpgid",
                "epoll_ctl",
                "setresgid",
                "open",
                "uname",
                "mprotect",
                "sendmsg",
                "setsid",
                "sendfile",
                "fdatasync",
                "getcwd",
                "unlink",
                "epoll_pwait",
                "getpeername",
                "read",
                "arch_prctl",
                "rt_sigaction",
                "ppoll",
                "futex",
                "sendto",
                "lchown",
                "fchmodat",
                "flock",
                "eventfd",
                "dup",
                "setresuid",
                "munmap",
                "getsockname",
                "stat",
                "symlink",
                "getrusage",
                "pipe",
                "readlink",
                "getuid",
                "mkdir"
            ]
        }
    ]
}

added the following to my docker command;

docker create \
  --name=nextcloud \
  -e PUID=1000 \
  -e PGID=1000 \
  -e TZ=Australia/Melbourne \
  -p 3002:80 \
  -v $HOME/nextcloud/config:/config \
  -v $HOME/nextcloud/data:/data \
  --restart unless-stopped \
  --cap-drop=all \
  --cap-add=CHOWN \
  --cap-add=FOWNER \
  --cap-add=SETUID \
  --cap-add=SETGID \
  --cap-add=DAC_OVERRIDE \
  --security-opt seccomp:nextcloud-seccomp.json \
  linuxserver/nextcloud:17.0.1-ls52@sha256:5b928942971bca3ce30e012c8fa8694c41180b7f6e91804d5d271ace7c9085a6

and running with that profile errors out almost immediately with if: fatal: unable to spawn ifelse: Operation not permitted and i can't figure out why..

[aptalca]

On mobile, so I'll just add a couple of quick comments.

We use s6 overlay as the supervisor. It needs root access for the init processes. abc is the arbitrary user we use to run the unprivileged processes with inside the container. It's really a placeholder user name as it gets mapped to the PUID/PGID used in environment variables.

[udartsev]

1. scan your linux system for word kdevtmpfsi in files:
   sudo find / -type f -exec grep -l "kdevtmpfsi" {} +
2. You will see smth like:
   /var/lib/docker/volumes/d2076917e9f44b55d1fbfe2af6aca706f3cc52ca615e5f5de1ae1fdb4a040154/_data/kinsingvhBv0aQnc2
3. Delete that file:
   rm /var/lib/docker/volumes/d2076917e9f44b55d1fbfe2af6aca706f3cc52ca615e5f5de1ae1fdb4a040154/_data/kinsingvhBv0aQnc2
4. Reboot server:
   reboot

P.S.: Doesn`t works.

[aptalca]

Can you guys provide more clarification?

Are you suggesting that nextcloud is responsible for infecting your host with a crypto miner?

Did the OP run nextcloud on an already infected host?

Check the Crontab entries in nextcloud. If your host is compromised, potentially something may have modified the Crontab to run other scripts

[chrisdlangton]

My "host" is KVM. I actually run docker with gVisor runtime, not default runc which is probably not important here.
I say this only to point out that while the infection vector was nextcloud, my remediation involved destroying the "host" (KVM guest).

Now using nextcloud with restricted capabilities seems to solve the infection vector, the seccomp profile might also be a good safeguard if someone can figure out that ifelse call.

I think the biggest concern here is nextcloud had/has a software bug that this container has/had not fixed with a patch yet. In my research I noticed that no cve existed on Nextcloud itself but there are always bugs that don't get a CVE like the November PHP CSRF bug that specifically works on Nextcloud itself. You can see a PoC on exploitdb.

Basically the least privileged and defence in depth principals are your best solution as usual.

• Limit capabilities, whitelist with capadd
• Use a VM guest as a docker host
• Harden the docker host, lynis is a great tool here.
• apply patches regularly, and perhaps asap if the patch is a security update
• consider the gVisor runtime
• seccomp is a pain but probably one of the best defensive mechanisms we have for docker

[aptalca]

I'm sorry but I'm not quite following the logic here. The only link to nextcloud seems to be kdevtmpfsi running after starting nextcloud. That alone does not prove nextcloud was the infection vector, but it could as well be just an infection target (as we cannot replicate that on our systems).

You said i replicated the behavior, there is definitely a security hole in linuxserver/nextcloud docekr containr., but I'm not sure what methodology was used.

What we provide here is nextcloud on a pretty barebones alpine linux with nginx and php7, running as an unprivileged user (abc). All the config we ship is pretty much what nextcloud suggests (apart from not enabling HSTS, which imho is a sledgehammer and since our image does not expose port 80 to begin with, it is not crucial to enable). We also provide weekly image updates with OS package updates.

We had one recent nginx/php CVE that could potentially affect Nextcloud and patched that the same day php was fixed (I personally PRed the php update to alpine repo 3.10 branch).

If there is a vulnerability, it is likely upstream and would affect all nextcloud installs. But we (and they) would need more specific info to even start troubleshooting.

[chrisdlangton]

Agree with everything you said, the vulnerability is unknown to me and only found 1 know unpatched Nextcloud CSRF vulnerability that stems from Nextcloud use of some PHP. Like many vulnerabilities linked to PHP it's only the applications implementation choice that makes the vulnerability an issue any linuxserver can't do anything to patch that particular problem.

Also, that 1 CSRF thing I found on exploitdb still might not even be the vector I personally experienced.

To clarify your question about my replication. It was as simple a replication as I had already described.. the KVM guest which is the docker host, at the time if initial infection, had multiple containers. I isolatiled each container into a KVM guest of their own so that when Nextcloud 's KVM became infected the only processes running belonged to linuxserver/nextcloud container therefore the infection was a container breakout.

I've now hardened my KVM guest (the docker host), started using gVisor runtime, applied whitelist approach to capabilities, and for several other containers I was able to generate a seccomp profile using sysdig without any issue but hit a problem with ifelse on linuxserver/nextcloud.

I think we can probably close this issue.

Let's update the docs with advice to apply capabilities as I described above. Also if you figure out a seccomp profile that would be ideal

[freehussain]

i have the same issue here, after while i found this is related to supervisord service which run php-fpm command.
my solution is to stop docker container that run php-fpm