runtime: add credit system for scavenging

When scavenging small amounts it's possible we over-scavenge by a
significant margin since we choose to scavenge the largest spans first.
This over-scavenging is never accounted for.

With this change, we add a scavenge credit pool, similar to the reclaim
credit pool. Any time scavenging triggered by RSS growth starts up, it
checks if it can cash in some credit first. If after using all the
credit it still needs to scavenge, then any extra it does it adds back
into the credit pool.

This change mitigates the performance impact of golang.org/cl/159500 on
the Garbage benchmark. On Go1 it suggests some improvements, but most of
that is within the realm of noise (Revcomp seems very sensitive to
GC-related changes, both postively and negatively).

Garbage: https://perf.golang.org/search?q=upload:20190131.5
Go1:     https://perf.golang.org/search?q=upload:20190131.4

Performance change with both changes:

Garbage: https://perf.golang.org/search?q=upload:20190131.7
Go1:     https://perf.golang.org/search?q=upload:20190131.6

Change-Id: I87bd3c183e71656fdafef94714194b9fdbb77aa2
Reviewed-on: https://go-review.googlesource.com/c/160297
Reviewed-by: Austin Clements <[email protected]>

Author: mknyszek

src/runtime/mheap.go

@@ -107,6 +107,14 @@ type mheap struct {
 	// This is accessed atomically.
 	reclaimCredit uintptr
 
+	// scavengeCredit is spare credit for extra bytes scavenged.
+	// Since the scavenging mechanisms operate on spans, it may
+	// scavenge more than requested. Any spare pages released
+	// go to this credit pool.
+	//
+	// This is protected by the mheap lock.
+	scavengeCredit uintptr
+
 	// Malloc stats.
 	largealloc  uint64                  // bytes allocated for large objects
 	nlargealloc uint64                  // number of large object allocations
@@ -165,7 +173,7 @@ type mheap struct {
 	// simply blocking GC (by disabling preemption).
 	sweepArenas []arenaIdx
 
-	_ uint32 // ensure 64-bit alignment of central
+	// _ uint32 // ensure 64-bit alignment of central
 
 	// central free lists for small size classes.
 	// the padding makes sure that the mcentrals are
@@ -1349,6 +1357,14 @@ func (h *mheap) freeSpanLocked(s *mspan, acctinuse, acctidle bool, unusedsince i
 // starting from the largest span and working down. It then takes those spans
 // and places them in scav. h must be locked.
 func (h *mheap) scavengeLargest(nbytes uintptr) {
+	// Use up scavenge credit if there's any available.
+	if nbytes > h.scavengeCredit {
+		nbytes -= h.scavengeCredit
+		h.scavengeCredit = 0
+	} else {
+		h.scavengeCredit -= nbytes
+		return
+	}
 	// Iterate over the treap backwards (from largest to smallest) scavenging spans
 	// until we've reached our quota of nbytes.
 	released := uintptr(0)
@@ -1377,6 +1393,10 @@ func (h *mheap) scavengeLargest(nbytes uintptr) {
 		h.scav.insert(s)
 		released += r
 	}
+	// If we over-scavenged, turn that extra amount into credit.
+	if released > nbytes {
+		h.scavengeCredit += released - nbytes
+	}
 }
 
 // scavengeAll visits each node in the unscav treap and scavenges the