runtime: panic on plan9_arm builders

[millerresearch]

The plan9_arm builders have been getting panics on every run since the afternoon of 27 Oct. The first one seems to be https://build.golang.org/log/6a299fffd128c3ed0bfdd1c471c2ca891dee8b34 after CL 232298 was merged.

The immediate cause of the panic varies. Could be a memory corruption.

[dmitshur]

This may be the same as or related to #42237.

[millerresearch]

This may be the same as or related to #42237.

I don't think so. Those examples are deliberate panics because of a timeout. The plan9_arm panics are unexpected signals triggered by something going weirdly wrong in the runtime.

[millerresearch]

In some cases, the immediate cause of the panic is that _g_.m.p is nil at the top of findrunnable(). Shouldn't that be impossible?

[ianlancetaylor]

CC @aclements @mknyszek @prattmic

[millerresearch]

Another common panic cause is that _g_.m is nil after calling acquirep() in mstart1(). Also impossible?

[prattmic]

Thanks for the heads up. I agree that this looks different from the failures in #42237, though perhaps it is related. Hopefully this one is reproducible on gomote.

[prattmic]

Does this builder have stability issues (not sure who maintains it)? After waiting 4hr 40min (!!!!) for a gomote, I tried to run some tests:

$ gomote run user-mpratt-plan9-arm-0 ./go/src/all.rc  
Building Go cmd/dist using /usr/glenda/go
Building Go toolchain1 using /usr/glenda/go.
2020/11/04 16:02:28 user-mpratt-plan9-arm-0: timeout after 10s waiting for headers for /status
2020/11/04 16:02:50 Buildlet https://farmer.golang.org:443 failed three heartbeats; final error: 502 Bad Gateway
Error running run: Error trying to execute ./go/src/all.rc: Buildlet https://farmer.golang.org:443 failed heartbeat after 86.211812ms; marking dead; err=502 Bad Gateway

And it seems to still be gone:

$ gomote ping user-mpratt-plan9-arm-0
Error running ping: 502 Bad Gateway

I notice on the build dashboard that the builds that don't have these panics contain errors like: communication error to buildlet (promoted to terminal error): network error promoted to terminal error: runTests: dist test failed: all buildlets had network errors or timeouts, yet tests remain, which sounds awfully similar.

I'm not thrilled by the prospect of waiting another 5hr to see if the next gomote will share a similar fate.

cc @golang/release

[millerresearch]

Does this builder have stability issues

Well, that's one way of putting it. As I said above, these builders have been crashing on every run since the 27 October CL I referred to. Sometimes they panic, sometimes the corruption is so bad that the go runtime just hangs. There isn't a full time attendant to keep restarting them manually.

I've been putting in some time trying to debug this, and the common factor seems to be that
_g_.m is nil or garbage. I don't know enough about the internals of the scheduler to understand how this could happen.

[prattmic]

I see at least one crash that does look immediately like bad g.m:

• https://build.golang.org/log/8135e1c551962cbe67aaa005110626dc6b6d7539: p.mcache is nil. This p came from pidleget, so it's not immediately coming off of a bad m (though it could be a downstream issue).

Overall, this looks like general memory corruption to me, but that it is limited to runtime internals is interesting. Perhaps bad TLS somehow?

[dmitshur]

not sure who maintains it

Builder owners and additional notes are available at https://farmer.golang.org/builders.

[millerresearch]

To corroborate the circumstantial evidence implicating CL 232298 I've tried patching the current head of the master branch with the reverse of commit 8fdc79e. (It needed a bit of hand editing because of later changes in runtime/proc.go)

With the commit reversed, all.rc completes with ALL TESTS PASSED, and no memory reference traps.

I'll run it a few more times, but it does appear that something in that commit has introduced a bug or tickled an existing bug whose effect is specific to plan9_arm.