runtime: test timeouts / deadlocks on NetBSD after CL 232298

[bcmills]

(Issue forked from #42237; see #42237 (comment).)

2020-11-11T06:26:05-f2e186b/netbsd-amd64-9_0
2020-11-10T18:42:47-8f2db14/netbsd-386-9_0
2020-11-10T18:33:37-b2ef159/netbsd-386-9_0
2020-11-10T18:33:37-b2ef159/netbsd-arm-bsiegert
2020-11-10T15:05:17-81322b9/netbsd-arm64-bsiegert
2020-11-09T21:03:36-d495712/netbsd-amd64-9_0
2020-11-09T20:02:56-5e18135/netbsd-386-9_0
2020-11-09T19:46:24-a2d0147/netbsd-386-9_0
2020-11-09T19:00:00-01cdd36/netbsd-386-9_0
2020-11-09T16:09:16-a444458/netbsd-386-9_0
2020-11-09T15:20:26-cb4df98/netbsd-386-9_0
2020-11-09T14:17:30-f858c22/netbsd-386-9_0
2020-11-07T16:59:55-5e371e0/netbsd-amd64-9_0
2020-11-07T16:31:02-2c80de7/netbsd-386-9_0
2020-11-07T03:19:27-33bc8ce/netbsd-386-9_0
2020-11-06T20:49:11-5736eb0/netbsd-amd64-9_0
2020-11-06T19:42:05-362d25f/netbsd-386-9_0
2020-11-06T15:33:23-d21af00/netbsd-386-9_0
2020-11-05T20:35:00-2822bae/netbsd-arm64-bsiegert
2020-11-05T16:46:56-04b5b4f/netbsd-386-9_0
2020-11-05T16:46:56-04b5b4f/netbsd-amd64-9_0
2020-11-05T15:16:57-34c0969/netbsd-386-9_0
2020-11-05T14:54:35-74ec40f/netbsd-386-9_0
2020-11-05T00:21:39-c018eec/netbsd-386-9_0
2020-11-04T21:45:25-fd841f6/netbsd-amd64-9_0
2020-11-04T16:54:48-594b4a3/netbsd-386-9_0
2020-11-04T15:53:19-5f0fca1/netbsd-386-9_0
2020-11-04T06:12:33-8eb846f/netbsd-amd64-9_0
2020-11-03T23:05:51-e1b305a/netbsd-386-9_0
2020-11-03T04:11:02-974def8/netbsd-386-9_0
2020-11-03T00:50:57-ebc1b8e/netbsd-amd64-9_0
2020-11-02T21:08:14-4fcb506/netbsd-386-9_0
2020-11-02T21:00:57-d1efaed/netbsd-arm64-bsiegert
2020-11-02T03:03:16-0387bed/netbsd-amd64-9_0
2020-11-01T13:23:48-0be8280/netbsd-arm64-bsiegert
2020-10-31T08:41:25-f14119b/netbsd-amd64-9_0
2020-10-31T00:35:18-79fb187/netbsd-arm64-bsiegert
2020-10-30T22:21:02-64a9a75/netbsd-arm-bsiegert
2020-10-30T21:14:09-f96b62b/netbsd-386-9_0
2020-10-30T21:14:09-f96b62b/netbsd-amd64-9_0
2020-10-30T21:02:17-420c68d/netbsd-386-9_0
2020-10-30T20:20:58-6abbfc1/netbsd-386-9_0
2020-10-30T18:06:13-6d087c8/netbsd-386-9_0
2020-10-30T18:05:53-36d412f/netbsd-amd64-9_0
2020-10-30T18:01:54-fb184a3/netbsd-amd64-9_0
2020-10-30T17:54:57-e02ab89/netbsd-amd64-9_0
2020-10-30T16:29:11-1af388f/netbsd-amd64-9_0
2020-10-30T16:20:05-2b9b272/netbsd-386-9_0
2020-10-30T16:20:05-2b9b272/netbsd-amd64-9_0
2020-10-30T00:23:50-faa4426/netbsd-386-9_0
2020-10-30T00:13:25-60f42ea/netbsd-386-9_0
2020-10-30T00:03:40-01efc9a/netbsd-386-9_0
2020-10-29T22:45:29-f588974/netbsd-amd64-9_0
2020-10-29T22:44:49-f43e012/netbsd-386-9_0
2020-10-29T21:46:54-fe70a3a/netbsd-386-9_0
2020-10-29T18:26:42-0b798c4/netbsd-amd64-9_0
2020-10-29T15:27:22-68e30af/netbsd-386-9_0
2020-10-29T15:13:09-50af50d/netbsd-amd64-9_0
2020-10-29T15:11:47-ecb79e8/netbsd-386-9_0
2020-10-29T13:53:33-c45d780/netbsd-amd64-9_0
2020-10-29T08:08:26-d9725f5/netbsd-386-9_0
2020-10-29T08:00:50-8b51798/netbsd-386-9_0
2020-10-29T03:23:51-15f01d6/netbsd-386-9_0
2020-10-29T01:50:09-308ec22/netbsd-386-9_0
2020-10-29T00:07:35-c1afbf6/netbsd-386-9_0
2020-10-28T17:54:13-fc116b6/netbsd-amd64-9_0
2020-10-28T17:10:08-642329f/netbsd-386-9_0
2020-10-28T17:08:06-e3c58bb/netbsd-amd64-9_0
2020-10-28T16:17:54-421d4e7/netbsd-386-9_0
2020-10-28T14:25:56-b85c2dd/netbsd-amd64-9_0
2020-10-28T13:25:44-72dec90/netbsd-386-9_0
2020-10-28T05:02:44-150d244/netbsd-386-9_0
2020-10-28T04:20:39-02335cf/netbsd-amd64-9_0
2020-10-28T01:03:23-368c401/netbsd-386-9_0
2020-10-27T23:12:41-5d3666e/netbsd-386-9_0
2020-10-27T22:13:30-091257d/netbsd-arm-bsiegert
2020-10-27T21:29:13-009d714/netbsd-amd64-9_0
2020-10-27T21:28:53-933721b/netbsd-amd64-9_0
2020-10-27T20:22:56-9113d8c/netbsd-amd64-9_0
2020-10-27T20:04:19-de2d1c3/netbsd-amd64-9_0
2020-10-27T20:03:41-5c1122b/netbsd-amd64-9_0
2020-10-27T20:03:12-79a3482/netbsd-386-9_0
2020-10-27T19:52:40-c515852/netbsd-amd64-9_0
2020-10-27T18:38:48-f0c9ae5/netbsd-amd64-9_0
2020-10-27T18:38:48-f0c9ae5/netbsd-arm64-bsiegert
2020-10-27T18:13:59-3f6b1a0/netbsd-amd64-9_0

[bcmills]

2020-10-14T08:05:58-fc3a6f4/netbsd-amd64-9_0

has similar symptoms from before CL 232298, but it's the only such occurrence, and the failure rate is markedly higher since then.

[bcmills]

(CC @prattmic @aclements @ChrisHines @bsiegert @coypoop)

[bsiegert]

/cc @tklauser

This issue together with #42422 means that the NetBSD builders are much more flaky than we would like :(

[prattmic]

Still ongoing, as expected:

2020-11-30T17:45:06-c193279/netbsd-amd64-9_0
2020-11-27T20:31:33-91f77ca/netbsd-amd64-9_0
2020-11-27T20:31:33-91f77ca/netbsd-arm-bsiegert
2020-11-26T05:00:21-f0ff6d4/netbsd-amd64-9_0
2020-11-25T19:46:00-4481ad6/netbsd-amd64-9_0
2020-11-25T15:59:35-df68e01/netbsd-arm64-bsiegert
2020-11-25T02:51:30-1308f11/netbsd-386-9_0
2020-11-24T21:47:20-ba2adc2/netbsd-386-9_0
2020-11-24T07:07:06-6965b01/netbsd-386-9_0
2020-11-24T07:07:06-6965b01/netbsd-amd64-9_0
2020-11-24T03:06:15-7dc5d90/netbsd-386-9_0
2020-11-23T19:58:25-48a1a51/netbsd-amd64-9_0
2020-11-21T16:46:05-9ea6364/netbsd-386-9_0
2020-11-21T16:46:05-9ea6364/netbsd-arm64-bsiegert
2020-11-21T03:29:37-f93ef07/netbsd-386-9_0
2020-11-20T17:31:50-c306fd6/netbsd-386-9_0
2020-11-20T04:21:00-66c0264/netbsd-386-9_0
2020-11-20T02:27:53-0dcc7d6/netbsd-amd64-9_0
2020-11-20T00:09:05-7eed73f/netbsd-386-9_0
2020-11-19T20:37:38-59f5fda/netbsd-386-9_0
2020-11-19T20:37:38-59f5fda/netbsd-amd64-9_0
2020-11-19T19:30:38-e73697b/plan9-arm
2020-11-19T04:02:42-ff2824d/netbsd-386-9_0
2020-11-19T02:17:10-0bb6115/netbsd-386-9_0
2020-11-19T01:27:31-96b943a/netbsd-amd64-9_0
2020-11-18T22:12:57-5b0ec1a/netbsd-386-9_0
2020-11-18T20:12:03-b4f3d52/netbsd-386-9_0
2020-11-18T20:12:03-b4f3d52/netbsd-arm64-bsiegert
2020-11-18T19:09:36-ae76f6e/netbsd-amd64-9_0
2020-11-18T14:45:14-b194b51/netbsd-386-9_0
2020-11-18T12:04:06-399b5d1/netbsd-386-9_0
2020-11-18T04:43:50-bcfaeca/netbsd-386-9_0
2020-11-18T04:43:50-bcfaeca/netbsd-amd64-9_0
2020-11-17T18:28:55-01df2fe/netbsd-386-9_0
2020-11-17T15:10:45-0968d2d/netbsd-amd64-9_0
2020-11-17T13:20:20-0ae3b7c/netbsd-386-9_0
2020-11-16T22:24:14-869e295/netbsd-386-9_0
2020-11-16T18:39:54-38367d0/netbsd-386-9_0
2020-11-13T23:26:54-4f63e0a/netbsd-arm-bsiegert
2020-11-13T22:01:37-86954d5/netbsd-386-9_0
2020-11-12T22:50:40-30ba798/netbsd-386-9_0
2020-11-12T21:21:29-f016172/netbsd-amd64-9_0
2020-11-12T21:21:18-60b1253/netbsd-386-9_0
2020-11-12T14:17:24-e75aef8/netbsd-386-9_0
2020-11-12T14:17:24-e75aef8/netbsd-amd64-9_0
2020-11-12T13:40:55-4bc5f6f/netbsd-amd64-9_0
2020-11-11T16:22:54-2843754/netbsd-386-9_0
2020-11-11T16:22:54-2843754/netbsd-amd64-9_0
2020-11-11T06:26:05-f2e186b/netbsd-386-9_0

[prattmic]

This seems to be reproducible on gomote (~20% of the time) with:

$ gomote push ${MOTE?}
$ gomote run ${MOTE?} ./go/src/make.bash
$ gomote run ${MOTE?} ./go/bin/go test -count=1 -short -timeout=2m -run=TestNoShrinkStackWhileParking -trace ./trace.out -o ./runtime.test runtime

[prattmic]

This issue is reminding me a bit of hard-to-reproduce issues we've been tracking with worldsema for a while. That is, the test is getting work done, and then just ... stops:

Two minutes later forcegchelper runs and lo and behold, the test starts making progress again (G30):

Note necessarily related, but suspicious.

cc @aclements @mknyszek

[prattmic]

Update for the day: I've narrowed down the issue to findrunnable finding a timer to wait on, deciding to netpollBreak to kick the poller... and then nothing. Either netpollBreak isn't waking the poller or there isn't actually a poller.

Hopefully, I'll be able to narrow this further tomorrow.

[prattmic]

There is a poller, but it is not woken by netpollBreak. Here's the flow:

# netpoll with 180s delta (test timeout)
[0.753745874 P -1] M 4 P ? now 16465974762371 netpoll poll delay 179247420027

# Successfully write to netpoll break FD
[0.753788314 P -1] M 5 P ? netpollBreak: write =  1

# A different (non-blocking) netpoll receives the netpollBreak readiness, but doesn't clear netpollWakeSig
# because this is a non-blocking call: https://cs.opensource.google/go/go/+/master:src/runtime/netpoll_kqueue.go;l=149-156
[0.753803005 P 2] M 7 P ? now 16465974819894 netpoll poll delay 0
[0.753806724 P 2] M 7 P ? now 16465974823551 netpoll poll delay 0 duration 3657 n 1
[0.753808903 P 2] M 7 P ? now 16465974825645 netpoll received netpollBreak delay 0

# Later netpollBreak calls don't write to FD because netpollWakeSig = 1
[0.754030446 P 0] M 0 P ? netpollBreak wakeSig 1

# 180s later netpoll finally completes, does not receive the netpollBreak readiness (n = 1)
[180.253550882 P -1] M 4 P ? now 16645474571844 netpoll poll delay 179247420027 duration 179499809473 n 0

The 180s netpoll should have woken from netpollBreak but did not. I see two primary possibilities:

1. The NetBSD kernel is treating this event as edge-triggered with exactly-once delivery. It delivered a notification to the non-blocking netpoll (which ignored it), so it didn't deliver to the blocking call as well.
2. Delivery of this event is simply flaky in the kernel in general.

I suppose I'll take a look at the kernel source tomorrow...

[prattmic]

Or (3) the event is edge-triggered and the kqueue call was slightly too late. Note that the netpollBreak write is very close in time to the netpoll call (which I'm logging before the call).

[prattmic]

@bsiegert are you aware of any recent issues with the NetBSD kernel that may cause the behavior described in #42515 (comment)?

To add further details, all following calls to kevent after the netpollBreak are getting the break event returned, so possibility (1) doesn't seem right. (Note all these calls are non-blocking). It is just the blocking kevent call that occurred right around the same time as the pipe write that isn't getting an event returned.

The source all seems OK to me:

1. kqueue lock is taken.
2. If the event is already in the queue, then we should get it (though there is a lot of logic there).
3. If the event isn't in the queue, wait on the condition variable.

This seems fine, but makes me wonder if there is some race where the condition variable isn't firing via a racing broadcast. The kqueue lock seems to prevent this, but who knows.