runtime: "found pointer to free object" on linux-386 starting 2021-11-04

[bcmills]

greplogs --dashboard -md -l -e '(?ms)\Alinux-386.*found pointer to free object' --since=2021-10-01

2021-11-04T20:31:02-978e39e/linux-386-buster
2021-11-04T20:24:01-99699d1/linux-386-buster
2021-11-04T20:24:01-99699d1/linux-386-jessie
2021-11-04T20:24:01-99699d1/linux-386-softfloat

(CC @mknyszek, maybe?)

[mknyszek]

Ah! Yes... I think this is probably https://golang.org/cl/356609. I'll see if I can get any info, if not I'll prep a rollback.

I was digging into this already actually trying to reproduce locally because I saw it on the trybots for later CLs, but now thinking about it I bet it's really https://golang.org/cl/356609.

[mknyszek]

Actually... there are some segfaults on linux-amd64, too. I suspect these might have the same root cause; they appear correlated. The s390x builder is also failing, but very consistently, so I'm gonna try there.

If I look at dashboard logs and include the segfaults, the earliest crash I get is

2021-11-04T20:00:54-961aab2/linux-amd64-unified

which is when I set turned on the pacer redesign. Everything before that is e.g. plan9-arm.

[mknyszek]

Oh, wait. I think this is another GC-ran-during-a-test-that-it-didn't-before issue. The failure seems consistently in hammerStoreLoadPointer.

If I add a couple of runtime.GC calls in the test, while it's doing the hammering I'm able to reproduce. Fix incoming.

[mknyszek]

More evidence to this point: all the zombies appear to contain pointers that have the property v&((1<<16)-1) == v>>16 (or something similar, like for 32 bits).

[gopherbot]

Change https://golang.org/cl/361455 mentions this issue: sync/atomic: disable GC during TestHammerStoreLoad