runtime: flush write barrier buffer to create work

Currently, if the gcWork runs out of work, we'll fall out of the GC
worker, even though flushing the write barrier buffer could produce
more work. While this is not a correctness issue, it can lead to
premature mark 2 or mark termination.

Fix this by flushing the write barrier buffer if the local gcWork runs
out of work and then checking the local gcWork again.

This reduces the number of premature mark terminations during all.bash
by about a factor of 10.

Updates #26903. This is preparation for eliminating mark 2.

Change-Id: I48577220b90c86bfd28d498e8603bc379a8cd617
Reviewed-on: https://go-review.googlesource.com/c/134315
Run-TryBot: Austin Clements <[email protected]>
TryBot-Result: Gobot Gobot <[email protected]>
Reviewed-by: Rick Hudson <[email protected]>

Author: aclements

src/runtime/mgcmark.go

@@ -882,6 +882,13 @@ func gcDrain(gcw *gcWork, flags gcDrainFlags) {
 			b = gcw.tryGetFast()
 			if b == 0 {
 				b = gcw.tryGet()
+				if b == 0 {
+					// Flush the write barrier
+					// buffer; this may create
+					// more work.
+					wbBufFlush(nil, 0)
+					b = gcw.tryGet()
+				}
 			}
 		}
 		if b == 0 {
@@ -963,6 +970,12 @@ func gcDrainN(gcw *gcWork, scanWork int64) int64 {
 		b := gcw.tryGetFast()
 		if b == 0 {
 			b = gcw.tryGet()
+			if b == 0 {
+				// Flush the write barrier buffer;
+				// this may create more work.
+				wbBufFlush(nil, 0)
+				b = gcw.tryGet()
+			}
 		}
 
 		if b == 0 {