x/build/cmd/release: 2 targets are not passing tests successfully on master branch

[dmitshur]

The release command is used by releasebot to build targets. In the first stage (-mode=prepare), it runs tests. In second stage (-mode=release), it skips tests.

release supports 12 normal targets (not counting new optional longtest-only targets; described in #29252 but can be ignored in this issue), see here.

These targets are passing tests (they run successfully without -skip_tests flag given to release) on a recent master commit (e.g., 09e791f):

• src
• linux-armv6l
• linux-ppc64le
• linux-s390x

These are not passing:

• darwin-amd64 - blocked on x/build/cmd/release: fails for -target=linux-amd64 due to non-default GOROOT_FINAL value causing TestScript/test_race_install_cgo test to fail #39385 and x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391
• freebsd-386 - blocked on x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391 only
• freebsd-amd64 - blocked on x/build/cmd/release: fails for -target=linux-amd64 due to non-default GOROOT_FINAL value causing TestScript/test_race_install_cgo test to fail #39385 and x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391
• linux-386 - blocked on x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391 only
• linux-amd64 - blocked on x/build/cmd/release: fails for -target=linux-amd64 due to non-default GOROOT_FINAL value causing TestScript/test_race_install_cgo test to fail #39385 and x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391
• linux-arm64 - blocked on x/build/cmd/release: fails for -target=linux-amd64 due to non-default GOROOT_FINAL value causing TestScript/test_race_install_cgo test to fail #39385 and x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391
• windows-386 - blocked on x/build/cmd/release: TestScript/test_race_install_cgo and TestDWARF fail for -target=windows-amd64, and just TestDWARF fails for -target=windows-386 #39478 blocked on x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391 only
• windows-amd64 - blocked on x/build/cmd/release: TestScript/test_race_install_cgo and TestDWARF fail for -target=windows-amd64, and just TestDWARF fails for -target=windows-386 #39478 blocked on x/build/cmd/release: fails for -target=linux-amd64 due to non-default GOROOT_FINAL value causing TestScript/test_race_install_cgo test to fail #39385 and x/build/cmd/release: test/winbatch.go test fails for -target=darwin-amd64 #39391

Each one can be reproduced if your account has permissions needed to run releasebot (they are documented at https://github.com/golang/build/tree/master/cmd/releasebot#permissions) with:

$ release -target=<target_name> -version=go1.15beta1 -watch -rev=09e791feb1dfbfc05578a17cd9c035ef82af4033

(The release command creates a tarball and places it in a temporary staging directory; it doesn't publish anything, so it is safe to run as shown above.)

See #39385 as an example of a specific test that is failing.

This is the high-level tracking issue for the entire problem. Next, I'll be updating it and splitting into smaller, distinct failures that can be investigated independently.

This issue requires investigation. The problem is either in how the release command sets up the environment and executes commands on the target builder, or in the builder definition, or in the test, or some combination. Depending on what we learn, we may end up deciding this is okay to resolve after beta1 (but before the final release) and skip these test failures in order to make the Go 1.15 beta 1 available for testing and finding unknown issues sooner.

/cc @andybons @toothrot @cagedmantis @golang/osp-team

[dmitshur]

Both known issues #39385 and #39391 have been resolved by now, so I'm testing this again.

I'm using Go commit cacac8b (latest on master branch as of this posting) for this test.

By now, these targets have passed (to be more precise, got as far as failing the API check test, without other failures) at least once:

• src
• linux-armv6l
• linux-ppc64le
• linux-s390x
• darwin-amd64
• freebsd-386
• freebsd-amd64
• linux-386
• linux-amd64
• linux-arm64

However, windows-386 and windows-amd64 have failed:

windows-386

ok  	cmd/go	111.031s

--- FAIL: TestDWARF (1.28s)
    dwarf_test.go:39: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	25.084s

windows-amd64

go test proxy running at GOPROXY=http://127.0.0.1:50986/mod
go proxy: no archive rsc.io v1.5.2: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.1.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.5.1: file does not exist
go proxy: no archive example.com/newcycle v1.0.0: file does not exist
go proxy: no archive test.go v1.0.0: file does not exist
go proxy: no archive rsc.io v1.5.2: file does not exist
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_race_install_cgo (1.21s)
        script_test.go:210: 
            # Tests Issue #10500 (1.209s)
            > [!race] skip
            > [!darwin] ! stale cmd/cgo  # The darwin builders are spuriously stale; see #33598.
            FAIL: testdata\script\test_race_install_cgo.txt:5: cmd/cgo is unexpectedly stale
            
            
FAIL
FAIL	cmd/go	111.443s

--- FAIL: TestDWARF (1.62s)
    dwarf_test.go:39: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	19.749s

This may be related to the change in CL 236819. /cc @bcmills

I'll try again a few more times to confirm if it's a flaky or reproducible failure. I'll also try the windows targets with a commit before CL 236819 to see what effect it has. Then update this issue.

[bcmills]

That one is strange because those two tests are both passing on the regular builders. I'll pop on a gomote and see if they're easy to reproduce with GOROOT_FINAL.

[dmitshur]

I've tried both windows-amd64 and windows-386 again, and both failed in the same way, so this doesn't seem to be a flake. Filed #39478 as a dedicated tracking issue for it. Thank you again for looking into this @bcmills.

[dmitshur]

Issue #39478 has been resolved via CL 237359; a re-test of all targets with it is now ongoing.

[dmitshur]

Testing has shown no new test failures. All 12 targets are passing tests when targeting commit 2a4e71c (CL 237359). That means this issue is resolved. Huge thanks to @bcmills for helping make this happen!