runtime: scavenger pacing fails to account for fragmentation [1.13 backport]

[gopherbot]

@mknyszek requested issue #34048 to be considered for backport to the next 1.13 minor release.

Since this has the potential to cause a severe performance loss, I think we should backport this to Go 1.13.

@gopherbot Please open a backport issue for 1.13.

[bcmills]

this has the potential to cause a severe performance loss

@mknyszek, @aclements: could you describe the conditions under which this performance degradation occurs? (Is it predictable? Does it show up during testing? Is there a workaround available? Do we have a rough estimate for the fraction of users affected?)

[mknyszek]

Is it predictable?

Yes. If the application's span fragmentation (heap_inuse / heap_alloc) >= 10%, the scavenger can spiral into a state in which every free span is scavenged, leading to a large number of page faults and a large number of syscalls.

Does it show up during testing?

I don't have a specific test, but I can recreate the case with both large and small programs. I'd have to write a short program which reproduces the problem, but that shouldn't be difficult to do.

Is there a workaround available?

No. There is no way to work around this because it's embedded in runtime pacing. You'd have to somehow keep your application's span fragmentation under 10% which isn't reasonable to ask of anyone.

Do we have a rough estimate for the fraction of users affected?

No, and I'm not sure exactly how we'd be able to get that. We know for a fact that it severely affects Kubernetes' API latency and is one reason why they're blocked on moving to Go 1.13 (see #32828). There is another (medium-sized, I suppose?) Google application which took consistently took 80% longer to run in Go 1.13 as a direct consequence of this issue. The patch in the non-backport bug fixes this behavior, and brings it back to Go 1.12 levels of performance.

[bcmills]

By “Is it predictable?” and “Does it show up during testing?”, I meant more, “will users be able to predict and/or identify that they are affected?”

(That is: this seems more important to backport if it can crop up suddenly after a user has already tested and deployed a new build.)

[mknyszek]

Ah, I think I understand. Then it's not predictable. It can certainly show up suddenly in production, without anyone noticing while running tests, running microbenchmarks, or even a smaller version of the full application. It's as difficult to predict as how much heap fragmentation your application produces, which may be a function of input in some cases.

[aclements]

I agree that this should be backported for the reasons @mknyszek laid out.

Specifically, the CL to backport is https://golang.org/cl/193040.