runtime: "found bad pointer in Go heap" on darwin/arm64

[bcmills]

runtime: pointer 0x14000270000 to unallocated span span.base()=0x14000266000 span.limit=0x14000271d00 span.state=0
runtime: found in object at *(0x14000130050+0x8)
object=0x14000130050 s.base()=0x14000130000 s.limit=0x14000131fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0x140006d4257
 *(object+8) = 0x5 <==
 *(object+16) = 0x16f923847
 *(object+24) = 0x7
 *(object+32) = 0x14000026e20
 *(object+40) = 0x1c
 *(object+48) = 0x1400002a07f
 *(object+56) = 0x10
 *(object+64) = 0x1
 *(object+72) = 0x1
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x10097fa76?, 0x6?})
	/tmp/buildlet/go/src/runtime/panic.go:992 +0x50 fp=0x16fe96d40 sp=0x16fe96d10 pc=0x10050eda0
runtime.badPointer(0x107e885e0, 0x14000035238?, 0x14000130050, 0x16fe96e10?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:368 +0x120 fp=0x16fe96d90 sp=0x16fe96d40 pc=0x1004efaf0
runtime.findObject(0x14000780f60?, 0x1005158e0?, 0x140001024e0?)
	/tmp/buildlet/go/src/runtime/mbitmap.go:410 +0xb4 fp=0x16fe96dd0 sp=0x16fe96d90 pc=0x1004efc54
runtime.scanobject(0x14000130050, 0x14000035238)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1316 +0x150 fp=0x16fe96e60 sp=0x16fe96dd0 pc=0x1004fa8c0
runtime.gcDrain(0x14000035238, 0x3)
	/tmp/buildlet/go/src/runtime/mgcmark.go:1081 +0x1c8 fp=0x16fe96ec0 sp=0x16fe96e60 pc=0x1004fa0b8
runtime.gcBgMarkWorker.func2()
	/tmp/buildlet/go/src/runtime/mgc.go:1276 +0x98 fp=0x16fe96f10 sp=0x16fe96ec0 pc=0x1004f6f98
runtime.systemstack()
	/tmp/buildlet/go/src/runtime/asm_arm64.s:237 +0x6c fp=0x16fe96f20 sp=0x16fe96f10 pc=0x10053c23c

greplogs --dashboard -md -l -e '(?ms)\Adarwin-arm64.*fatal error: found bad pointer in Go heap'

2021-12-08T17:24:46-ac7e950/darwin-arm64-12_0-toothrot
(Note two-year gap!)
2019-09-26T17:34:54-430d2aa/darwin-arm64-corellium
2019-09-25T16:33:28-8189a06/darwin-arm64-corellium
2019-08-29T19:54:46-75198b9/darwin-arm64-corellium
2019-08-27T21:02:43-cc6feab/darwin-arm64-corellium
2019-07-03T07:14:40-e2fdce9/darwin-arm64-corellium
2019-06-26T20:10:05-fc26cba/darwin-arm64-corellium

[bcmills]

Note that darwin/arm64 is a first class port.

CC @jeremyfaller for routing.

[cherrymui]

Before the "gap" the darwin-arm64-corellium was the iOS builder. They may not be related.

[cherrymui]

Bad click. No intention to close.

[bcmills]

Before the "gap" the darwin-arm64-corellium was the iOS builder. They may not be related.

Agreed. (I'm much more concerned about this new one, because I think it's more likely to reflect a source of crashes on real users' hardware.)

[cherrymui]

From the message it seems the bad pointer is 0x14000270000, found at *(0x14000130050+0x8), but later on

*(object+8) = 0x5 <==

It looks like the memory content is changing between the two reads, so there is something concurrently writing to the object. But 5 is also not a valid pointer value. So from the other writer's perspective this field should not be a pointer?

[cherrymui]

This may be interesting, or may be completely red-herring:

The object's content looks a lot like some frames' arguments on some of the stacks:

 *(object+0) = 0x140006d4257
 *(object+8) = 0x5 <==
 *(object+16) = 0x16f923847
 *(object+24) = 0x7
 *(object+32) = 0x14000026e20
 *(object+40) = 0x1c
 *(object+48) = 0x1400002a07f
 *(object+56) = 0x10
 *(object+64) = 0x1
 *(object+72) = 0x1

goroutine 262

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4261, 0x3}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 264

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4271, 0x6}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 135

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4151, 0x7}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

goroutine 266

cmd/go/internal/load.loadPackageData({0x100b3abe8, 0x14000028160}, {0x140006d4291, 0x7}, {0x16f923847, 0x7}, {0x14000026e20, 0x1c}, {0x1400002a07f, 0x10}, ...)

object+16 and later words match exactly with the third and later arguments. object+0 and object+8 are also quite close to the second arguments. However, the object's address, 0x14000130050, doesn't match any of the stacks, and it is on an mSpanInUse span instead of a mSpanManual.

[randall77]

@cherrymui Could it be a layout object (stackArgs?) used by reflect.Call?

[cherrymui]

That function is not called by reflect call, so probably not.

[randall77]

Could also be a closure, I guess.

[cherrymui]

That call is at https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/load/pkg.go;l=1028
It is in a closure but the closure would capture parent, instead of parent.ImportPath, parent.Dir, parent.Root and other arguments in that order. The order of the fields in the bad object also doesn't match the parent object.

[ianlancetaylor]

This has not happened again. Optimistically closing.