runtime: fatal error: malloc deadlock on s390x

[Vishwanatha-HD]

What version of Go are you using (go version)?

go version - "devel go1.21-f36dc54e9c" on linux s390x architecture.

Does this issue reproduce with the latest release?

Yes. The issue is reproducible easily.

What operating system and processor architecture are you using (go env)?

go env Output

GO111MODULE=""
GOARCH="s390x"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="s390x"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/vishwa/Workspaces/golang/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/vishwa/Workspaces/golang/go/pkg/tool/linux_s390x"
GOVCS=""
GOVERSION="devel go1.21-f36dc54e9c Thu Feb 23 06:44:23 2023 +0000"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/vishwa/Workspaces/golang/go/src/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -march=z196 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3776423934=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Race detection feature is not working properly on linux s390x. In order to evaluate that, I ran the "race.bash" script from "go/src" directory and I see that the script fails, and this is one of the errors that I am seeing.

What did you expect to see?

I expected the "race.bash" to run completely without any errors or issues.

What did you see instead?

The stack trace of the error is as below:

fatal error: malloc deadlock
panic during panic

goroutine 55 [running]:
runtime.throw({0x1293c20, 0xf})
        /home/vishwa/Workspaces/golang/go/src/runtime/panic.go:1075 +0x58 fp=0xc000121688 sp=0xc000121660 pc=0x1054228
runtime.mallocgc(0x30, 0x1279920, 0x1)
        /home/vishwa/Workspaces/golang/go/src/runtime/malloc.go:937 +0xb06 fp=0xc000121700 sp=0xc000121688 pc=0x10200b6
runtime.newobject(0x1279920)
        /home/vishwa/Workspaces/golang/go/src/runtime/malloc.go:1254 +0x3a fp=0xc000121728 sp=0xc000121700 pc=0x102029a
os.newProcess(...)
        /home/vishwa/Workspaces/golang/go/src/os/exec.go:29
os.startProcess({0xc00022a090, 0x15}, {0xc00020c120, 0x2, 0x2}, 0xc000121b30)
        /home/vishwa/Workspaces/golang/go/src/os/exec_posix.go:63 +0x75c fp=0xc000121868 sp=0xc000121728 pc=0x10e350c
os.StartProcess({0xc00022a090, 0x15}, {0xc00020c120, 0x2, 0x2}, 0xc000121b30)
        /home/vishwa/Workspaces/golang/go/src/os/exec.go:109 +0x7e fp=0xc0001218d0 sp=0xc000121868 pc=0x10e2cae
os/exec.(*Cmd).Start(0xc000214160)
        /home/vishwa/Workspaces/golang/go/src/os/exec/exec.go:693 +0xad4 fp=0xc000121b78 sp=0xc0001218d0 pc=0x1171de4
os/exec.(*Cmd).Run(0xc000214160)
        /home/vishwa/Workspaces/golang/go/src/os/exec/exec.go:587 +0x42 fp=0xc000121bb0 sp=0xc000121b78 pc=0x1171262
syscall_test.testAmbientCaps(0xc00023e4e0, 0x0)
        /home/vishwa/Workspaces/golang/go/src/syscall/exec_linux_test.go:633 +0x1462 fp=0xc000121e80 sp=0xc000121bb0 pc=0x11fd752
syscall_test.TestAmbientCaps(0xc00023e4e0)
        /home/vishwa/Workspaces/golang/go/src/syscall/exec_linux_test.go:526 +0x46 fp=0xc000121e98 sp=0xc000121e80 pc=0x11fc256
testing.tRunner(0xc00023e4e0, 0x12a0ee0)
        /home/vishwa/Workspaces/golang/go/src/testing/testing.go:1579 +0x244 fp=0xc000121fb0 sp=0xc000121e98 pc=0x115bd84
testing.(*T).Run.func1()
        /home/vishwa/Workspaces/golang/go/src/testing/testing.go:1632 +0x8e fp=0xc000121fd8 sp=0xc000121fb0 pc=0x115db8e
runtime.goexit()
        /home/vishwa/Workspaces/golang/go/src/runtime/asm_s390x.s:749 +0x2 fp=0xc000121fd8 sp=0xc000121fd8 pc=0x1096202
created by testing.(*T).Run in goroutine 1
        /home/vishwa/Workspaces/golang/go/src/testing/testing.go:1632 +0x894

[bcmills]

(attn @golang/s390x)

[mknyszek]

That error message comes up when malloc is reentered from within malloc, but that's clearly not happening in the stack trace. That suggests to us (here in triage) that it's actually memory corruption of some kind, otherwise in theory we would be seeing this failure elsewhere. The mechanism for checking this condition is very simple, so I doubt it's actually a problem with the mechanism itself, either.