x/build/cmd/coordinator: write proper scheduler

[bradfitz]

Currently if there are N builds waiting on a buildlet type (due to lack of machines or quota), the current implementation is a bunch of goroutines fighting over a mutex.

It's random who wins and gets the buildlet.

We should have them register interest and when a buildlet becomes available, pick the highest priority one.

Example priorities in order:

• trybot run with a +2 already
• trybot run
• normal run of a build at tip of a release branch
• normal run of a build at tip of master branch
• normal run of a build at tip of dev branch
• normal run of a build not at tip
• idle flake detection (x/build: auto-detect flakes #19177)

etc.

/cc @danp @kevinburke

[bradfitz]

And gomote access is in there somewhere too.

[gopherbot]

CL https://golang.org/cl/38306 mentions this issue.

[gopherbot]

CL https://golang.org/cl/40397 mentions this issue.

[gopherbot]

Change https://golang.org/cl/70430 mentions this issue: cmd/coordinator: attempt to run newer work before older work on buildlets