runtime: flush each idle P's page cache at the end of each GC cycle

Currently pages may linger in an idle P's page cache, hiding the memory
from the scavenger precisely when it's useful to return memory to the OS
and reduce the application's footprint.

Change-Id: I49fbcd806b6c66991d1ca87949f76a9f06708e70
Reviewed-on: https://go-review.googlesource.com/c/go/+/453622
Run-TryBot: Michael Knyszek <[email protected]>
TryBot-Result: Gopher Robot <[email protected]>
Reviewed-by: Michael Pratt <[email protected]>
Auto-Submit: Michael Knyszek <[email protected]>

Author: mknyszek

src/runtime/mgc.go

@@ -1104,9 +1104,21 @@ func gcMarkTermination() {
 	// mcache before allocating, but idle Ps may not. Since this
 	// is necessary to sweep all spans, we need to ensure all
 	// mcaches are flushed before we start the next GC cycle.
+	//
+	// While we're here, flush the page cache for idle Ps to avoid
+	// having pages get stuck on them. These pages are hidden from
+	// the scavenger, so in small idle heaps a significant amount
+	// of additional memory might be held onto.
 	systemstack(func() {
 		forEachP(func(pp *p) {
 			pp.mcache.prepareForSweep()
+			if pp.status == _Pidle {
+				systemstack(func() {
+					lock(&mheap_.lock)
+					pp.pcache.flush(&mheap_.pages)
+					unlock(&mheap_.lock)
+				})
+			}
 		})
 	})
 	// Now that we've swept stale spans in mcaches, they don't