cmd/compile: panic: offset too large

[elagergren-spideroak]

What version of Go are you using (go version)?

$ go version
go version go1.19 darwin/amd64

Does this issue reproduce with the latest release?

Unsure.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN="/Users/x/gopath/bin"
GOCACHE="/Users/x/Library/Caches/go-build"
GOENV="/Users/x/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/x/gopath/pkg/mod"
GONOPROXY="*.s.com,*.s-s.com"
GONOSUMDB="*.s.com,*.s-s.com"
GOOS="darwin"
GOPATH="/Users/x/gopath"
GOPRIVATE="*.s.com,*.s-s.com"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.19"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2b/74fz3jhd4wz4vnbf4z7ywzww0000gp/T/go-build668550611=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

$ PROJ_TEST_LOG_STDERR=1 go test -v -vet all -failfast -run TestStashOutgoing 2>&1 | tee /tmp/logs.txt
# spideroak.com/proj/policy [spideroak.com/proj/policy.test]
panic: offset too large

goroutine 560 [running]:
cmd/compile/internal/liveness.(*argLiveness).emit(0xc001039800)
	/usr/local/go/src/cmd/compile/internal/liveness/arg.go:310 +0x3aa
cmd/compile/internal/liveness.ArgLiveness(0xc001ec92c0, 0xc000c32540, 0x123e35c?)
	/usr/local/go/src/cmd/compile/internal/liveness/arg.go:218 +0xab5
cmd/compile/internal/ssagen.genssa(0xc000c32540, 0xc00098dce0)
	/usr/local/go/src/cmd/compile/internal/ssagen/ssa.go:6834 +0x117
cmd/compile/internal/ssagen.Compile(0xc001ec92c0, 0xc002712f90?)
	/usr/local/go/src/cmd/compile/internal/ssagen/pgen.go:193 +0x26f
cmd/compile/internal/gc.compileFunctions.func4.1(0x0?)
	/usr/local/go/src/cmd/compile/internal/gc/compile.go:153 +0x3a
cmd/compile/internal/gc.compileFunctions.func3.1()
	/usr/local/go/src/cmd/compile/internal/gc/compile.go:140 +0x4d
created by cmd/compile/internal/gc.compileFunctions.func3
	/usr/local/go/src/cmd/compile/internal/gc/compile.go:138 +0x78
FAIL	spideroak.com/proj/policy [build failed]

I don't have any sort of repro yet. I've built this project easily hundreds of times with Go 1.19 and this is the first time it's happened. Of course, it hasn't happened in the dozens of test runs since. I know it's not much to work with, but it's probably better to report ICEs than not, right?

[cherrymui]

I've built this project easily hundreds of times with Go 1.19 and this is the first time it's happened.

Were they building the same source code, or different? That is, is this a nondeterministic failure, or an ICE because the input source code is changed? Thanks.

[elagergren-spideroak]

@cherrymui sorry, usually different source code.

I've lost my scrollback buffer, so I can't say for certain, I believe this compiler panic occurred after making an edit to the source code. But, I did re-run that same source code after the compiler panic occurred and it did (and has not) not panicked.

[cherrymui]

Thanks. So it sounds like nondeterministic failure. Maybe next time if you can have some input that can trigger the failure (even nondeterministically), that would be helpful.

For that particular panic, I think we only add to lv.args for offset < 0xff https://cs.opensource.google/go/go/+/master:src/cmd/compile/internal/liveness/arg.go;l=106, so it shouldn't happen. Maybe memory corruption or data race?

[elagergren-spideroak]

Yeah, if I can reproduce this again I'll certainly post more information.

[mjl-]

I've been seeing the same issue a few times during the last few days, but with go1.19.2 on linux/amd64.

go build
go vet
staticcheck
go test -shuffle=on -coverprofile cover.out
# github.com/mjl-/bstore [github.com/mjl-/bstore.test]
panic: offset too large

goroutine 1473 [running]:
cmd/compile/internal/liveness.(*argLiveness).emit(0xc00540f800)
	/usr/local/go/src/cmd/compile/internal/liveness/arg.go:310 +0x3aa
cmd/compile/internal/liveness.ArgLiveness(0xc003893a40, 0xc005712700, 0xc000370690?)
	/usr/local/go/src/cmd/compile/internal/liveness/arg.go:218 +0xab5
cmd/compile/internal/ssagen.genssa(0xc005712700, 0xc0023f4e00)
	/usr/local/go/src/cmd/compile/internal/ssagen/ssa.go:6909 +0x117
cmd/compile/internal/ssagen.Compile(0xc003893a40, 0xc0052c4f90?)
	/usr/local/go/src/cmd/compile/internal/ssagen/pgen.go:197 +0x26f
cmd/compile/internal/gc.compileFunctions.func4.1(0xc000e0e030?)
	/usr/local/go/src/cmd/compile/internal/gc/compile.go:153 +0x3a
cmd/compile/internal/gc.compileFunctions.func3.1()
	/usr/local/go/src/cmd/compile/internal/gc/compile.go:140 +0x4d
created by cmd/compile/internal/gc.compileFunctions.func3
	/usr/local/go/src/cmd/compile/internal/gc/compile.go:138 +0x78
FAIL	github.com/mjl-/bstore [build failed]

$ go version
go version go1.19.2 linux/amd64

$ uname -a
Linux x1 5.18.0-4-amd64 #1 SMP PREEMPT_DYNAMIC Debian 5.18.16-1 (2022-08-10) x86_64 GNU/Linux

(This code isn't online yet.) It doesn't do anything special. It does use generics, unlike most Go code I compile much more often without getting this random panic. If it happens again, I'll try a go clean and a rebuild. For me too the next build after this panic (without changing any source files) succeeds as normal.

I'm assuming it is OK to see paths pointing to /usr/local even though I don't have any Go toolchain/source there. I keep my go toolchains in $HOME/sdk and that's where I can find the files with linenumbers from the stacktrace.

[mjl-]

I have been able to reproduce this on multiple two more machines with the same code and go1.19.2. Faster machines seem to get to the problem with fewer attempts. My laptop triggers it typically within 200 runs. My macbook pro 2016 darwin/amd64 gets there within 200 runs as well, I got it once over 420 runs on a slower linux/amd64 cloud vm with 2 processors. I have not been able to trigger this on an older linux/386 cloud vm with 1 processor after about 500 runs. On my laptop it didn't yet trigger after 520 runs with GOMAXPROCS=1, and also not after 700 runs with GO19CONCURRENTCOMPILATION=0 (a flag I've seen before, I don't know if this makes sense for this kind of error). My laptops /proc/cpuinfo lists 8 processors of Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz.

I have been getting these errors while running "go test", but I've change the _test.go files to regular .go files and just "go build" triggers the same error. I could send the code somewhere.

[cherrymui]

@mjl- thanks for the update. It would be great if you could post a reproducer. It is fine if it only fails with 200 runs, or even 2000 runs. I guess there is some sort of race, which is only triggered with some specific input source code.

Would you mind also try to see if it fails if you build the compiler with the race detector? That is, go install -race cmd/compile then go build (or go test) your program. (After that you may want to go install cmd/compile to restore the normal compiler, so it doesn't slow down all compilations.) Thanks.

[mjl-]

# github.com/mjl-/bstore
==================
WARNING: DATA RACE
Read at 0x00c001eaa100 by goroutine 679:
  cmd/compile/internal/ir.(*Name).FrameOffset()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ir/name.go:203 +0x1ff7
  cmd/compile/internal/liveness.ArgLiveness()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/liveness/arg.go:106 +0x2004
  cmd/compile/internal/ssagen.genssa()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/ssa.go:6909 +0x2da
  cmd/compile/internal/ssagen.Compile()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/pgen.go:197 +0x4c6
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:153 +0x53
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b

Previous write at 0x00c001eaa100 by goroutine 683:
  cmd/compile/internal/ir.(*Name).SetFrameOffset()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ir/name.go:204 +0x21b
  cmd/compile/internal/abi.(*ABIConfig).updateOffset()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/abi/abiutils.go:458 +0x1d2
  cmd/compile/internal/abi.(*ABIConfig).ABIAnalyze()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/abi/abiutils.go:441 +0x364
  cmd/compile/internal/ssagen.buildssa()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/ssa.go:458 +0x29fc
  cmd/compile/internal/ssagen.Compile()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/pgen.go:187 +0x5a
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:153 +0x53
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b

Goroutine 679 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:138 +0xd6
  cmd/compile/internal/gc.compileFunctions.func4()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:152 +0x63
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:154 +0xbb
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b

Goroutine 683 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:138 +0xd6
  cmd/compile/internal/gc.compileFunctions.func4()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:152 +0x63
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:154 +0xbb
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b
==================
==================
WARNING: DATA RACE
Read at 0x00c001eaa0ec by goroutine 679:
  cmd/compile/internal/ir.(*Name).Addrtaken()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ir/name.go:249 +0x44
  cmd/compile/internal/liveness.ArgLiveness.func1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/liveness/arg.go:119 +0x18
  cmd/compile/internal/liveness.ArgLiveness()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/liveness/arg.go:123 +0x3c5
  cmd/compile/internal/ssagen.genssa()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/ssa.go:6909 +0x2da
  cmd/compile/internal/ssagen.Compile()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/pgen.go:197 +0x4c6
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:153 +0x53
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b

Previous write at 0x00c001eaa0ec by goroutine 683:
  cmd/compile/internal/ir.(*bitset16).set()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ir/bitset.go:35 +0x284
  cmd/compile/internal/ir.(*Name).SetIsOutputParamInRegisters()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ir/name.go:261 +0x259
  cmd/compile/internal/abi.(*ABIConfig).updateOffset()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/abi/abiutils.go:459 +0x22a
  cmd/compile/internal/abi.(*ABIConfig).ABIAnalyze()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/abi/abiutils.go:441 +0x364
  cmd/compile/internal/ssagen.buildssa()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/ssa.go:458 +0x29fc
  cmd/compile/internal/ssagen.Compile()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/ssagen/pgen.go:187 +0x5a
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:153 +0x53
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b

Goroutine 679 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:138 +0xd6
  cmd/compile/internal/gc.compileFunctions.func4()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:152 +0x63
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:154 +0xbb
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b

Goroutine 683 (running) created at:
  cmd/compile/internal/gc.compileFunctions.func3()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:138 +0xd6
  cmd/compile/internal/gc.compileFunctions.func4()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:152 +0x63
  cmd/compile/internal/gc.compileFunctions.func4.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:154 +0xbb
  cmd/compile/internal/gc.compileFunctions.func3.1()
      /home/mjl/sdk/go1.19.2/src/cmd/compile/internal/gc/compile.go:140 +0x5b
==================
Found 2 data race(s)

I'll upload the code in a moment.

[cherrymui]

Thanks! It looks to me like that two functions shared a single ir.(*Name), perhaps for a parameter? (Maybe one of them is a wrapper or some compiler-generated function?) This seems wrong.

[mjl-]

Code is at https://www.ueber.net/who/mjl/tmp/bstore.gorace.tgz
It is about 7k lines, so hardly a minimal reproducer... The test_*.go files were the tests. Seems like compiling them stresses the compiler more.
With the compiler-with-race-detector a race is detected in just few runs for me.

[cuonglm]

@mjl- to be clear, how could we reproduce this? I mean running go test or anything else?

I couldn't reproduce in my Linux.