runtime: randomize scheduler for testing, perhaps in -race mode #11372
Description
Trying Go 1.5 inside Google we've found a number of tests that implicitly depend on the order in which goroutines run. Before Go 1.5 if you started 5 goroutines they ran in order 1,2,3,4,5. Now the scheduler prefers the last readied goroutine so they run in order 5,1,2,3,4. The failures remind me quite a bit of the map dependency failures we had before we randomized map iteration.
It would be nice to shake these out more systematically, like random map iteration does for map order bugs. Random execution is probably not good for general performance, but I wonder if we should do it in some kind of testing mode. I thought about doing it for _test binaries, or maybe by default from TestMain, but then you have to make sure you turn it off for benchmarks, and its not clear you want to start coloring tests differently in this way.
Maybe it makes sense to make running with -race mode randomize the scheduler? My limited understanding is that this shouldn't affect the races detected (although if it did that might help as much as it might hurt). And if you are running with -race you probably care more about shaking out bugs than about raw performance, so it's a plausible signal for making the scheduler a bit less predictable.
A colleague suggested this paper as worth reading.
http://research.microsoft.com/apps/pubs/default.aspx?id=118655
(A Randomized Scheduler with Probabilistic Guarantees of Finding Bugs, Madanlal Musuvathi, Sebastian Burckhardt, Pravesh Kothari, and Santosh Nagarakatte, ASPLOS 2010)