runtime: 1.23 hangs when running under qemu-user [bisected]

[cheese]

Go version

go version go1.23.3 linux/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/root/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/root/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/go/pkg/tool/linux_arm64'
GOVCS=''
GOVERSION='go1.23.3'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/root/.config/go/telemetry'
GCCGO='gccgo'
GOARM64='v8.0'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2732334604=/tmp/go-build -gno-record-gcc-switches'

What did you do?

When I build Go 1.23 with the same version of Go under qemu-user, the build hangs all the way. The issue can be reproduced with a Dockerfile:

from quay.io/fedora/fedora:41

RUN dnf install -y @development-tools
RUN curl -O https://go.dev/dl/go1.23.3.src.tar.gz && curl -O https://go.dev/dl/go1.23.3.linux-arm64.tar.gz
RUN tar fx go1.23.3.linux-arm64.tar.gz && mkdir src && tar fx go1.23.3.src.tar.gz  -C src
RUN cd src/go/src/; env PATH=$PATH:/go/bin bash make.bash

Steps:

1. Setup qemu-user binfmt for a foreign ISA, for example, installs qemu-user-static-aarch64 on Fedora.
2. Build the Dockerfile for specified arch: podman build --arch aarch64 .

Tested on Fedora 40 with 6.9.8-200.fc40.x86_64 and Fedora 41 with 6.11.6-300.fc41.x86_64.

Tested with qemu-user 9.1.1 of riscv64/loongarch64/aarch64.

What did you see happen?

The go processes of the build hangs forever:

                 │ ├─crun-buildah-buildah663882417.scope
                 │ │ └─container
                 │ │   ├─912459 /usr/bin/qemu-aarch64-static /usr/bin/bash make.bash
                 │ │   ├─914348 /usr/bin/qemu-aarch64-static ./cmd/dist/dist bootstrap -a
                 │ │   ├─914395 /usr/bin/qemu-aarch64-static /go/bin/go install "-tags=math_big_pure_go compiler_bootstrap purego" bootstrap/cmd/...
                 │ │   └─914743 /usr/bin/qemu-aarch64-static /go/bin/go install "-tags=math_big_pure_go compiler_bootstrap purego" bootstrap/cmd/...

The issue does not exist with Go 1.22. I did a bisect and found the offending commit d068c2c .

What did you expect to see?

Build of Go 1.23 runs through under qemu-user.

[gabyhelp]

Related Issues

• runtime: build fails when run via QEMU for linux/amd64 running on linux/arm64 #69255
• runtime: go sometimes hangs under qemu-arm-static since 1.21 #67355 (closed)
• go build hangs forever in 1.21.x #64705 (closed)
• runtime: various crashes running ARM 'go' command under qemu #29325 (closed)
• runtime: multi-arch build via qemu fails to exec go binary #68976 (closed)
• Go 1.17+ crashes on Docker Desktop (M1) when running with "--platform=linux/amd64" platform #48439 (closed)
• runtime: async preemption support on ARM64 breaks ARM64 on QEMU #36981
• go build hangs forever in 1.22.x #68582 (closed)
• runtime: ppc64x binaries randomly segfault on linux 5.13rc6 #46803 (closed)
• cmd/link: regression on building an app with `GOOS=wasip1 GOARCH=wasm` #65786 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[cherrymui]

Please note that qemu-user is not really a supported problem. As it is caused by CL 560615, it is probably due to a qemu bug or inconsistency in how it handles kernel APIs.

cc @panjf2000 @ianlancetaylor for CL 560615

[cheese]

Cross-posted to qemu: https://gitlab.com/qemu-project/qemu/-/issues/2738

[cheese]

Limiting GOMAXPROCS to a small number can relieve this issue.