x/build/cmd/coordinator: misc/cgo/testplugin.TestIssue25756pie fails on darwin-arm64 builder deterministically but only in sharded mode

[cherrymui]

CL https://go-review.googlesource.com/c/go/+/319489 added a test to misc/cgo/testplugin. The test is failing in the darwin-arm64 builder.

Interestingly, I cannot reproduce the failure locally on a darwin-arm64 machine, by running cd GOROOT/misc/cgo/testplugin; go test. I also cannot reproduce the failure on gomote, by running gomote run $VM go/bin/go tool dist test testplugin or gomote run $VM go/src/all.bash.

But it does fail consistently on the build dashboard and with trybot ( https://storage.googleapis.com/go-build-log/d4df70bf/darwin-arm64-11_0-toothrot_5af7c061.log )

Maybe there is some unusual setting on the builder?

cc @toothrot @cagedmantis @dmitshur

[dmitshur]

This test has also passed for me locally on an M1 machine with macOS 11.3.1, so it seems this being a builder issue is likely. I'll try to look at why it's failing in post-submit but not via gomote.

[dmitshur]

The test also passes on the builder as invoked during release tests (i.e., with something like release -target=darwin-arm64 -version=go1.17beta123 -watch -rev=ff7d5f97b3375a87e2de90f42e96983e5f0f95a4). This supports a theory that the problem could be related to coordinator using SplitMakeRun mode (where it compiles, then snapshots, then runs tests) during TryBots and post-submit builds.

[bcmills]

This supports a theory that the problem could be related to coordinator using SplitMakeRun mode

That suggests a possible connection to #33598.

[dmitshur]

I've investigated this more today. The test is passing on a physical darwin/arm64 machine, via sequential (non-sharded) test execution, when executed on its own, but failing specifically when executed by cmd/coordinator in sharded test mode on this builder. It's really nice that it fails so reproducibly and gives us a chance to investigate it, as it's likely the root cause for this problem may shared across more builder issues, though the feedback loop is quite slow.

Getting to the bottom of it will take longer, so I'll send a CL to skip the test for now, so that its failure doesn't mask other darwin/arm64 regressions in the mean time.

[gopherbot]

Change https://golang.org/cl/321349 mentions this issue: misc/cgo/testplugin: skip TestIssue25756pie on darwin/arm64 builder

[dmitshur]

Based on investigation above, this is a bug in the build system rather than a test failure. The problematic builder is skipped via CL 321349, so this is no longer blocking Go 1.17 release.

I think this is a good opportunity for us to investigate further, likely after the 1.17 release, so retitling and moving to next milestone.