docker buildx run yum install very slow, while docker build is fine

[light4]

docker buildx run yum install very slow, while docker build is fine

build with these commands bellow, docker build takes about 3min to finish, while docker buildx build takes about 3 hours.

sudo docker build -f centos7.Dockerfile .

sudo docker buildx build -f centos7.Dockerfile --target=artifact --output type=local,dest=$(pwd)/rpms/ .

I think docker buildx should use docker build caches, but it's not. This is unexpected.

centos7.Dockerfile

FROM centos:centos7 as build
LABEL maintainer="opsdev@qunar.com"

RUN yum install -y yum-utils rpm-build redhat-rpm-config make gcc git vi tar unzip rpmlint wget curl \
    && yum clean all

# 安装 golang
RUN PKG_VERSION="1.15.1" PKG_NAME="go$PKG_VERSION.linux-amd64.tar.gz" \
    && wget https://dl.google.com/go/$PKG_NAME \
    && tar -zxvf $PKG_NAME -C /usr/local \
    && rm -rf $PKG_NAME

ENV GOROOT=/usr/local/go
ENV GOPATH=/home/q/go
ENV PATH=$PATH:/usr/local/go/bin:/home/q/go/bin
ENV GOPROXY=https://goproxy.io

# RUN useradd q -u 5002 -g users -p q
# USER q
ENV HOME /home/q
WORKDIR /home/q
RUN mkdir -p /home/q/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
RUN echo '%_topdir %{getenv:HOME}/rpmbuild' > /home/q/.rpmmacros

COPY spec/q-agentd.spec q-agentd.spec
COPY scripts/q-agentd.service /home/q/rpmbuild/SOURCES/q-agentd.service

RUN yum-builddep -y q-agentd.spec \
    && rpmbuild -bb q-agentd.spec

FROM scratch as artifact
COPY --from=build /home/q/rpmbuild/RPMS/x86_64/*.rpm /

FROM build as release

[dubrsl]

The same problem

[darkdragon-001]

I also have the same problem. While using buildkit by default for some time via DOCKER_BUILDKIT=1 environment variable, CentOS 7 is the only distro where I experienced these problems with. I found out that yum is taking full CPU on a single core/thread (12.5% on my Intel CPU with 4 cores 8 threads).

Docker 20.10.6 on Fedora 33 on NVMe SSD.

[sxiii]

I happen to have same problem. Process goes on, but taking mostly 1 core and takes veeeeeeery long. Any leads on that?
Docker 20.10.5 (API version: 1.41, Go version: go1.13.15) on Manjaro Linux 21.0.5 kernel 5.11.19-1-MANJARO.

[silh]

Similar problem with go get.

[x2c3z4]

Same issue, any update?

[clime]

The same problem here.

[Diluka]

not only yum install, but also npm install or any other large amount of file IO manipulations

for example, 10 minutes for buildx to build 2 platforms, and 2 minutes each for build to.

[clime]

Ye, I would probably try to use buildkit but this is a blocker for me.

[slimm609]

I recently ran into this issue as well.

check the systemd service for docker.

It was set to infinity which sets the ulimit to 1073741816 but this seems to be causing a problem for some reason.

LimitNOFILE=infinity

When I changed the limit for nofile to 1024000, it resolved the problem and it works like normal docker build

LimitNOFILE=1024000

you can check the ulimit with a build

FROM alpine:latest
RUN echo "ulimit is $(ulimit -n)"

[mihalicyn]

Thanks @slimm609!

I've workarounded it by adding ulimit -n 1024000 just before yum install ... in the Dockerfile. It's more convenient rather than changing docker daemon service configuration at least in my case.

Example:

RUN ulimit -n 1024000 && yum -y install flex bison make gcc findutils openssl-devel bc diffutils elfutils-devel perl vim openssl dwarves

Hope this helps someone.

[Diluka]

try this https://gitlab.com/gitlab-org/gitlab-runner/-/issues/1698#note_15746725