runtime: failures in TestCtrlHandler with "could not read stdout: EOF" on windows-arm64

[bcmills]

greplogs --dashboard -md -l -e 'FAIL: TestCtrlHandler'

2021-11-08T17:46:34-2e210b4/windows-arm64-10
2021-11-07T04:56:11-85493d5/windows-arm64-10
2021-11-05T22:53:55-3b7e376/windows-arm64-10
2021-11-05T22:30:17-b07c41d/windows-arm64-10
2021-11-05T21:34:10-bb53fd7/windows-arm64-10
2021-11-04T21:50:21-bfd74fd/windows-arm64-10
2021-11-04T21:40:51-2c32f29/windows-arm64-10
2021-11-04T20:01:10-9b2dd1f/windows-arm64-10

[bcmills]

Marking as release-blocker since this appears to be a very recent regression (CC @aclements @mknyszek).

[mknyszek]

CC @bufflig

[bufflig]

Yea, this looks new and bad. I'll take it.

[bufflig]

@mknyszek - I wonder if you have thoughts on this?

It's reproducible beginning with the enabling of the new Pacer. It is fairly uncommon, but if you e.g. add a runtime.Gosched() after signal.Notify in the test program (which, of course should change nothing), it happens reliably but only on arm64.

Anyway, even when it happens, the signal walks through our handlers and we end up thinking we delivered the signal (sigsend in sigqueque.go), the sig.state is sigReceiving both when it works and when it doesn't reach the channel in the test program, and notewakeup is ending up in semawakeup. So everything happens as expected, but we never get anything on the channel (or the m does not get woken up). This happens with a low probability if the program is untouched, and a much higher probability if i yield after calling signal.Notify. If I disable the new pacer, I cannot get it to fail at all.

Could it be that the m does not get woken up in time? The process sits idle for a while and then it gets forcefully killed, which is also a little mysterious, but it does not seem to crash. It's possible taskkill eventually teminates it. Then the actual test fails as it gets EOF on the pipe.

[mknyszek]

Hm... The only thing the new pacer changes is when GC's happen, basically (in theory). Does GODEBUG=gctrace=1 indicate a GC happening where it doesn't with the pacer off? The GC in general perturbs the scheduling of goroutines quite a bit.

[bufflig]

You're spot on. That seems to be the problem. What happens is, I'm pretty sure, this:

• When a "signal" arrives to our ConsoleCtrlHandler, and it is a signal that would terminate the process if we returned (which is exactly what we're sending here), we post the signal (through another p) and then the (arbitrary) thread that handled the signal goes to sleep (real sleep, like stdcall1(_Sleep, uintptr(_INFINITE)))
• If a GC starts here and decides to preempt alll p's it will try to preempt the m that is now blocked by calling preemptM from the preeptone function.
• But on arm64, preemptMSupported is not true, and there's a TODO: Implement call injection in the beginning of preemptM(). So, nothing happens, we continue to wait for non-preempted p:s to proclaim themselves idle forever
• Meanwhile, the world is stopped, so the program is essentially one big idle-loop.

This happens when a GC is needed between the signal has arrived and the signal is processed by the little test program, and would not happen on amd64 as preemptM is implemented there. For unknown reasons, GC sometimes happen early and there'n no neeed for GC between the signal arrives and the program is terminated in a normal way, sometimes a GC is needed in the "bad" situation.

I see three solutions:

1. just disable the test on arm64/windows until the TODO can be addressed
2. Both disable the test and also make sure to not try to handle terminate-signals on arm64, as it will inevitably be flaky
3. implement preemptM on arm64, which I'm unsure of how hard it would be

I suggest 2 for 1.18, 3 for 1.19. Any input is welcome.

[cherrymui]

Implementing preemptM shouldn't be hard. But I agree we probably don't do it in the freeze.

[aclements]

Somewhat related, I have this CL for preemption on windows/arm, which I apparently failed to get in again and which probably needs a non-trivial rebase at this point.

[bufflig]

That is for 32bit arm though? I think the adding of arm64 conflicts with it, but as I just looked at the code, I can confirm that it would be a simple rebase. I could try to extend it to arm64. Still, I feel it would be best done in 1.19, but I could change my mind if you think it's better to do it now.