os/signal: timeout in TestAllThreadsSyscallSignals

[bcmills]

It's not clear to me whether this is a deadlock, a livelock, or just a slow test, but the similarity to #43149 is concerning (CC @AndrewGMorgan @ianlancetaylor).

panic: test timed out after 3m0s

goroutine 5 [running]:
testing.(*M).startAlarm.func1()
	/workdir/go/src/testing/testing.go:1701 +0xcc
created by time.goFunc
	/workdir/go/src/time/sleep.go:180 +0x44

goroutine 1 [chan receive]:
testing.(*T).Run(0xc00009c300, 0x17407a, 0x1c, 0x179b58, 0xf)
	/workdir/go/src/testing/testing.go:1240 +0x280
testing.runTests.func1(0xc00009c180)
	/workdir/go/src/testing/testing.go:1512 +0x78
testing.tRunner(0xc00009c180, 0xc0000a7d58)
	/workdir/go/src/testing/testing.go:1194 +0xd8
testing.runTests(0xc0000b4018, 0x259a60, 0x12, 0x12, 0xc000d45ed867509a, 0x29e8e2495d, 0x25d820, 0x1705ec)
	/workdir/go/src/testing/testing.go:1510 +0x2b4
testing.(*M).Run(0xc0000c6000, 0x0)
	/workdir/go/src/testing/testing.go:1418 +0x1a0
main.main()
	_testmain.go:81 +0x130

goroutine 18 [runnable]:
syscall.runtime_doAllThreadsSyscall(0xc00008a4d0)
	/workdir/go/src/runtime/proc.go:1669 +0x384
syscall.AllThreadsSyscall(0xab, 0x8, 0x0, 0x0, 0xdcc10, 0x229ed0, 0x7a32c)
	/workdir/go/src/syscall/syscall_linux.go:1064 +0xa8
os/signal.TestAllThreadsSyscallSignals(0xc00009c300)
	/workdir/go/src/os/signal/signal_linux_test.go:23 +0x40
testing.tRunner(0xc00009c300, 0x179b58)
	/workdir/go/src/testing/testing.go:1194 +0xd8
created by testing.(*T).Run
	/workdir/go/src/testing/testing.go:1239 +0x264
FAIL	os/signal	180.104s

2021-02-09T18:40:13-e9c9683/linux-ppc64-buildlet

[bcmills]

Tentatively milestoned to Go 1.16 because the function under test is new (https://tip.golang.org/doc/go1.16#syscall).

[ceseo]

@laboger FYI

[AndrewGMorgan]

Please assign this to me. I have no idea what might have happened here, yet, but I'll try to figure it out.

Looking at the build.golang.org status page, is this a one-off failure?

[ianlancetaylor]

This does seem to have occurred only once.

[AndrewGMorgan]

Quickly looking at the code https://tip.golang.org/src/os/signal/signal_linux_test.go it looks like the test was evaluating whether or not it should be skipped at the time of the 3 minute timeout.

Capturing the following just in case it otherwise gets lost at some point.

linux-ppc64-buildlet at e9c96835971044aa4ace37c7787de231bbde05d9

:: Running /workdir/go/src/make.bash with args ["/workdir/go/src/make.bash"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=ppc64_01" "GO_BUILDER_ENV=host-linux-ppc64-osu" "DEBIAN_FRONTEND=noninteractive" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GO_BUILD_KEY_DELETE_AFTER_READ=true" "GO_BUILD_KEY_PATH=/buildkey/gobuildkey" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=500ms" "GO_STAGE0_DL_DELAY=800ms" "WORKDIR=/workdir" "GO_BUILDER_NAME=linux-ppc64-buildlet" "GO_BUILDER_FLAKY_NET=1" "GOROOT_BOOTSTRAP=/usr/local/go-bootstrap" "GOBIN=" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "GOROOT_BOOTSTRAP=/usr/local/go-bootstrap"] in dir /workdir/go/src

Building Go cmd/dist using /usr/local/go-bootstrap. (go1.8 linux/ppc64)
Building Go toolchain1 using /usr/local/go-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/ppc64.
---
Installed Go for linux/ppc64 in /workdir/go
Installed commands in /workdir/go/bin
[... see failure at top of bug for end of this log...]

[AndrewGMorgan]

#42178 (comment) concerning the ppc64 build supports the detail that this code is CGO_ENABLED=0. My recollection of working on resolving that issue was that this architecture was noticeably slower overall than the systems I typically work with. This lends some support for a timeout being more likely on this architecture. However, the test should be nowhere near 3 minutes of runtime.

That being said, the code in this crash trace does not appear to have timed out while running the loop inside the test, but before that loop even starts and in the code after all the syscall parts on all the threads have run and just as code is trying to unstop the world (and reenable GC).

I've not yet reproduced the failure, and the build servers appear not to have failed again. But I'm still investigating.

[bcmills]

@AndrewGMorgan, another theory to consider: perhaps the actual slowdown was in some other os/signal test, and there simply wasn't enough time remaining, and TestAllThreadsSyscallSignals just happened to be the one of the remaining tests that was running when the timeout passed.

[AndrewGMorgan]

Did this ever repeat on the build servers?

Running against HEAD, the whole os/signal (nocgo) test normally completes in about 8.5 seconds. So, a 3 minute timeout suggests something very wrong happened.

Compressed into one bug update, hammering on this test for many hundreds of iterations, with some print() instrumentation, I have been able to reproduce the failure on the build server (linux-ppc64-buildlet-0)!

It's elusive and I'm not convinced it reproduces in the same place every time (different print()s are completing before the panic strikes, but I have also found that too many print()s can make it clear up - compiler subtlety?)... All the lockups, however, do seem to be within the startTheWorldGC() call at the very end of the syscall_runtime_doAllThreadsSyscall() function.

I'm using this command to reproduce the failure (it happens after a few attempts in a row, and a pass takes about 23s, so a 40s timeout seems fair):

~/go/bin/gomote run user-${USER}-linux-ppc64-buildlet-0 go/bin/go test os/signal -test.run TestAllThreadsSyscall -test.count=1000 -test.timeout=40s 

I'll keep trying to narrow it down, but we might want to get someone more familiar with startTheWorldGC() etc to take a crack at it. I'm less and less certain the issue is with the actual thread fixup support - which is the novel part of syscall_runtime_doAllThreadsSyscall()...

[bcmills]

Did this ever repeat on the build servers?

Turns out it did! Just not on ppc64:
2021-02-23T15:04:12-ab331c0/linux-amd64-nocgo