runtime: mlock of signal stack failed: 12

[karalabe]

What version of Go are you using (go version)?

$ go version
go version go1.14rc1 linux/amd64

Does this issue reproduce with the latest release?

I hit this with golang:1.14-rc-alpine docker image, the error does not happen in 1.13.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build968395959=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Clone https://github.com/ethereum/go-ethereum, replace the builder version in Dockerfile to golang:1.14-rc-alpine (or use the Dockerfile from below), then from the root build the docker image:

$ docker build .

FROM golang:1.14-rc-alpine

RUN apk add --no-cache make gcc musl-dev linux-headers git

ADD . /go-ethereum
RUN cd /go-ethereum && make geth

What did you expect to see?

Go should run our build scripts successfully.

What did you see instead?

Step 4/9 : RUN cd /go-ethereum && make geth
 ---> Running in 67781151653c
env GO111MODULE=on go run build/ci.go install ./cmd/geth
runtime: mlock of signal stack failed: 12
runtime: increase the mlock limit (ulimit -l) or
runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+
fatal error: mlock failed

runtime stack:
runtime.throw(0xa3b461, 0xc)
	/usr/local/go/src/runtime/panic.go:1112 +0x72
runtime.mlockGsignal(0xc0004a8a80)
	/usr/local/go/src/runtime/os_linux_x86.go:72 +0x107
runtime.mpreinit(0xc000401880)
	/usr/local/go/src/runtime/os_linux.go:341 +0x78
runtime.mcommoninit(0xc000401880)
	/usr/local/go/src/runtime/proc.go:630 +0x108
runtime.allocm(0xc000033800, 0xa82400, 0x0)
	/usr/local/go/src/runtime/proc.go:1390 +0x14e
runtime.newm(0xa82400, 0xc000033800)
	/usr/local/go/src/runtime/proc.go:1704 +0x39
runtime.startm(0x0, 0xc000402901)
	/usr/local/go/src/runtime/proc.go:1869 +0x12a
runtime.wakep(...)
	/usr/local/go/src/runtime/proc.go:1953
runtime.resetspinning()
	/usr/local/go/src/runtime/proc.go:2415 +0x93
runtime.schedule()
	/usr/local/go/src/runtime/proc.go:2527 +0x2de
runtime.mstart1()
	/usr/local/go/src/runtime/proc.go:1104 +0x8e
runtime.mstart()
	/usr/local/go/src/runtime/proc.go:1062 +0x6e

...
make: *** [Makefile:16: geth] Error 2

[josharian]

That is the consequence of trying to work around a kernel bug that significantly impacts Go programs. See #35777. The error message suggests the only two known available fixes: increase the ulimit or upgrade to a newer kernel.

[karalabe]

The error message suggests the only two known available fixes: increase the ulimit or upgrade to a newer kernel.

Well, I'm running the official alpine docker image, the purpose of which is to be able to build a Go program. Apparently it cannot. IMHO the upstream image should be the one fixed to fulfill its purpose, not our build infra to hack around a bug in the upstream image.

[josharian]

Is the Alpine image maintained by the Go team? (Genuine question. I don’t know about it.) Either way, yes, the image should be fixed, ideally with a kernel upgrade.

[karalabe]

I'm not fully sure who and how maintains the docker images (https://hub.docker.com/_/golang), but the docker hub repo is an "Official Image", which is a super hard to obtain status, so I assume someone high enough the food chain is responsible.

[networkimprov]

It's "maintained by the Docker Community". Issues should be filed at

https://github.com/docker-library/golang/issues

EDIT: the problem is the host kernel, not the Docker library image, so they can't fix it.

[karalabe]

So, the official solution to Go crashing is to point fingers to everyone else to hack around your code? Makes sense.

[josharian]

@karalabe I would like to remind you of https://golang.org/conduct. In particular, please be respectful and be charitable.

[karalabe]

Please answer the question

[josharian]

It is standard practice to redirect issues to the correct issue tracking system.

There is an extensive discussion of possible workarounds and fixes in the issue I linked to earlier, if you would like to see what options were considered on the Go side.

[karalabe]

This issue does not happen with Go 1.13. Ergo, it is a bug introduced in Go 1.14.

Saying you can't fix it and telling people to use workarounds it is dishonest, because reverting a piece of code would actually fix it. An alternative solution would be to detect the problematic platforms / kernels and provide a fallback mechanism baked into Go.

Telling people to use a different kernel is especially nasty, because it's not as if most people can go around and build themselves a new kernel. If alpine doesn't release a new kernel, there's not much most devs can do. And lastly if your project relies on a stable infrastructure where you can't just swap out kernels, you're again in a pickle.

It is standard practice to redirect issues to the correct issue tracking system.

The fact that Go crashes is not the fault of docker. Redirecting a Go crash to a docker repo is deflection.

[networkimprov]

You could also disable preemptive scheduling at runtime

$ GODEBUG=asyncpreemptoff=1 ./your_app

@ianlancetaylor we have a suggestion to do this when running on an affected kernel; is that viable?

BTW, It's a known problem that Docker library modules don't get timely updates, which is a security liability. Caveat emptor.

[josharian]

The kernel bug manifested as random memory corruption in Go 1.13 (both with and without preemptive scheduling). What is new in Go 1.14 is that we detect the presence of the bug, attempt to work around it, and prefer to crash early and loudly if that is not possible. You can see the details in the issue I referred you to.

Since you have called me dishonest and nasty, I will remind you again about the code of conduct: https://golang.org/conduct. I am also done participating in this conversation.

[networkimprov]

@karalabe, I misspoke, the issue is your host kernel, not the Docker image. Are you unable to update it?