Performing an image build with RUN adduser -u 1000000000 will result in no space left on host

[gbraad]

Description

Doing an adduser -u 1000000000 in a RUN will cause the filesystem on the host to fill up and report a No space on device left.

Steps to reproduce the issue:

1. Create a Dockerfile with RUN adduser user -u 1000000000 -g 0 -r -m -d /home/user/ -c "Default Application User"
2. Perform docker build -t test .
3. Wait until no space left on device is shown

Describe the results you received:

Step 3 : RUN adduser user -u 1000000000 -g 0 -r -m -d /home/user/ -c "Default Application User"
 ---> Running in e8472dbea9bf
ApplyLayer exit status 1 stdout:  stderr: write /var/log/lastlog: no space left on device
ERROR: Build failed: exit code 1

Describe the results you expected:
Succeeding build

Additional information you deem important (e.g. issue happens only occasionally):
Reproduced on GItlab CI runner as https://gitlab.com/gbraad/issue-docker/builds/6767776 (Source Dockerfile can be found in the repo: https://gitlab.com/gbraad/issue-docker)

Output of docker version:

Client:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.centos.10.x86_64
 Go version:      go1.6.3
 Git commit:      d381c64-unsupported
 Built:           Thu Aug  4 13:21:17 2016
 OS/Arch:         linux/amd64

Server:
 Version:         1.10.3
 API version:     1.22
 Package version: docker-common-1.10.3-46.el7.centos.10.x86_64
 Go version:      go1.6.3
 Git commit:      d381c64-unsupported
 Built:           Thu Aug  4 13:21:17 2016
 OS/Arch:         linux/amd64

Output of docker info:

Containers: 38
 Running: 0
 Paused: 0
 Stopped: 38
Images: 71
Server Version: 1.10.3
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.10.0-327.22.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 1
Total Memory: 489.1 MiB
Name: vps13.novalocal
ID: ATDQ:VMVE:SNUV:P67X:KPBK:DHHF:JFII:O23Q:VPVU:HV6I:6BZO:FE36
Username: gbraad
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Registries: docker.io (secure)

Additional environment details (AWS, VirtualBox, physical, etc.):

[AkihiroSuda]

Confirmed that this is still an issue in master 1ef10c8.

reproduction: curl https://gitlab.com/gbraad/issue-docker/raw/master/Dockerfile | docker build -
docker info:

Containers: 2
 Running: 0
 Paused: 0
 Stopped: 2
Images: 332
Server Version: 1.14.0-dev
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: 2k42xdn0j840958vyndm2jm8i
 Is Manager: true
 ClusterID: yk5quoro08hfi7sa31orgc2ph
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.146.0.2
 Manager Addresses:
  0.0.0.0:2377
  10.146.0.2:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 51371867a01c467f08af739783b8beafc154c4d7
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.69 GiB
Name: ws01
ID: SN3T:CNK6:JKQD:54CY:XKF2:BRX3:CIRU:DQBT:6DVZ:VWQ2:Q5ET:F23L
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 28
 Goroutines: 127
 System Time: 2016-11-29T07:10:46.152785794Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: true
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

[AkihiroSuda]

When I hit the issue, /var/lib/docker/overlay2/876025c2a053c0f8ccc8ceaeb8d4a508e31493277513c287672444c24e19a1dd/diff/var/log/lastlog grews up extremely large (filled with zero bytes).

Probably the issue is a bug of the implematation of adduser used in Fedora, rather than a bug of Docker itself?

However, when I ran adduser ... in docker run -it fedora:25, I cannot hit the issue.

[AkihiroSuda]

Closing as a dupe of #5419 (comment)

The underlying issue is that a large sparse file is created (approximately 32 GB), but it's not exactly a Docker bug.

[AkihiroSuda]

As mentioned in #5419 (comment), adduser -l should work

[gbraad]

@AkihiroSuda I am not sure how severe this is... although adduser -l will fix this, but as an automated build on Docker Hub this will result in the same error: https://hub.docker.com/r/gbraad/issue-dockerfile/builds/bsdmngknf8dhfreh8sgfmyu/

[rhatdan]

Just doing some hacking with info I found on the internet. Built a script that checks for sparse files.

cat /usr/bin/sparse
#!/bin/sh
sparse() {
	file=$1
	if [ "$((`stat -c '%b*%B-%s' -- "$file"`))" -lt 0 ]; then
	    echo "$file" is sparse
	fi
}
while IFS='' read -r file || [[ -n "$file" ]]; do
      sparse "$file"
done

# find /var/log -type f ! -size 0 | sparse
/var/log/lastlog is sparse

Could we change docker commit to not save sparse files and just zero them out?

[AkihiroSuda]

We would need to wait for golang/go#13548 to implement that.
(Or we can fork the tar pkg)

[rhatdan]

Last action on that was 9 Months ago. :^(

[runcom]

@rhatdan seems like someone commented on that pr to get a fix in by go 1.9 (more likely 1.10)

[gbraad]

Let's hope so, as this can cause issues with automated build processes.