internal/fuzz: "panic: unmarshaling fuzz input value after call: cannot unmarshal empty string"

[bcmills]

greplogs --dashboard -md -l -e 'panic: unmarshaling fuzz input value after call:' --since=2021-09-01

2021-09-30T18:10:18-205640e/windows-arm64-10
2021-09-29T16:45:42-2fad7db/windows-amd64-race

[katiehockman]

/cc @golang/fuzzing

[katiehockman]

It looks like this is still occurring. Last time it occurred was 4 days ago (https://build.golang.org/log/b5e821bc571fbcb208e626adc15cca8cd0a9a036)

I'm only seeing this error on Windows

[rolandshoemaker]

In two days I've managed to trigger this once, but that was before I had a chance to add any instrumentation, so not entirely sure what is happening.

cc @bufflig for windows knowledge. I suspect what is happening is the memory region is being zero'd (or shifted in some way between the processes) such that after writing something to it in one process in the other process it appears empty. None of the documentation I read for CreateFileMapping and MapViewOfFile indicated we were doing anything that could trigger this (perhaps something with the commit levels?) but entirely possible I missed something obvious.

Link to the relevant memory mapping code in question: https://go.googlesource.com/go/+/refs/heads/master/src/internal/fuzz/sys_windows.go#20

[gopherbot]

Change https://golang.org/cl/359354 mentions this issue: internal/fuzz: use atomics to synchronize shared memory read/write

[rolandshoemaker]

This turns out to be a file read/write race.

When multiple workers operate on the same input, it is possible that multiple workers find the same new input, returning it to the coordinator. It is also possible to process two identical inputs such that they provide different coverage maps, due to inherent flakiness in how we implement counters.

In the described case CoordinateFuzzing and workerClient.fuzz can race on writing the interesting input and reading the cached input, respectively. This can cause the read in workerClient.fuzz to return an empty slice, triggering the unmarshaling panic. This appears to trigger more frequently on Windows, due to some difference in how file concurrent writes/reads are handled.

[gopherbot]

Change https://golang.org/cl/359755 mentions this issue: internal/fuzz: don't add duplicate corpus entries