proposal: runtime: add way for applications to respond to GC backpressure

[josharian]

This is marked as a proposal, but I'd like it to start out as a discussion. Although I believe the need is clear, the correct design is not at all obvious.

Many applications have caches that they could easily drop in response to GC backpressure. However, there is currently no good way to do this.

One obvious example is sync.Pool. If we also had per-P storage, we could rebuild sync.Pool outside the standard library using GC backpressure. Better, folks could customize the sync.Pool emptying strategy in response to their own needs.

Another example is interning strings.

As to design, there are a few aspects. What does the API look like? When and how does the runtime trigger it? And what system dynamics might result.

One API ideas:

package runtime

// The API below needs better names. And the docs are a mix of user-facing and internal details.

// GCBackpressureC returns a channel that the runtime will close to indicate GC backpressure.
// One the channel has been closed, another call to GCBackpressureC will return a new channel.
// There is only a single, shared channel across all calls to GCBackpressureC.
// This makes the runtime side cheap: a single, lazily allocated channel, and a single close call per round of backpressure.
// The downside is that it could cause a stampede of memory releases.
// Another downside is that there could be logical races in which clients miss backpressure notifications by not obtaining a new channel quickly enough.
func GCBackpressureC() <-chan struct{}

// GCBackpressureC returns a channel that the runtime will send values on to indicate GC backpressure.
// The values have no meaning here, but maybe there's something useful we could send?
// The runtime side here is more expensive. It has O(n) channels to keep track of, and does O(n) work to send signals to everyone.
// On the other hand, by not sending signals to everyone simultaneously it can help avoid stampedes.
func GCBackpressureC() <-chan struct{}

As to when the runtime should indicate backpressure, there are several options. It could do it once per GC cycle, as a matter of course. (This matches existing sync.Pool behavior.) It could do it in response to heap growth. Or maybe something else or some combination.

It's unclear what effect this might have on system dynamics. To avoid messy situations like #22950, someone should probably do some hysteresis and/or incremental work. It is unclear whether that should reside in the runtime or in user code.

cc @aclements @rsc @dvyukov @dsnet @cespare

[ianlancetaylor]

See also #16843.

[josharian]

Thanks, Ian. Looks like the primary difference between this request and that one is that #16843 seems focused on dealing with heap limits, whereas I mostly had in mind improving the management of long-lived shared caches.

Another obvious use case is monitoring and metrics.

Another observation: you don’t want the runtime to block on these. So for the queue-per-consumer model you probably need it to be best effort only and use a buffered channel. Also, for the queue-per-consumer model, you could use the sent value to provide information about the memory event (GC cycle? Regular backpressure? > 50% on our way to heap growth? > 90% on our way to heap growth? Etc.)

cc @RLH

[CAFxX]

@josharian the closest thing I managed to do is https://github.com/CAFxX/gcnotifier, but it fires a notification at the end of a GC cycle, not when it starts (as is the case of the clearpools hook).

[dvyukov]

One thought about the interface until it's too late:

func GCBackpressureC() <-chan struct{}

Async interface based on chans can limit its usefulness. It won't be possible to understand when the callbacks have actually finished to do the scanning. The previous round may not have finished when we start another one. Sync callbacks may be more useful in preventing heap growth.

[aclements]

My big question is what does "GC backpressure" mean? In the context of SetMaxHeap, this is fairly clear: the GC is under pressure when it switches from the amortized proportional regime to the heap-limited regime. But outside of that context, the garbage collector is always under the same amount of pressure, by design.

If the point is to release caches, what you really want to know is what the trade-off is between using that memory for your caches versus giving the GC less memory to mark. I have no idea how you compute the trade-off.

In the case of sync.Pool, the current behavior leads to weird GC dynamics and I'm not sure I want to let users construct that sort of behavior without any runtime visibility into what's going on.

Related to @dvyukov's comment, it's not clear when you actually invoke this signal, but maybe defining backpressure would clarify that. Calling it when GC starts is too late for that GC cycle to collect anything that gets released. So you have to call it before GC starts, but how long before? You probably don't want to block GC until it's done (that would cause either deadlocks or unbounded heap growth). You could call it right after marking finishes, when the GC knows the targets for the next GC, but maybe that's too early.

FWIW, we've been thinking of splitting up SetMaxHeap and its notification channel because almost everyone who's tried SetMaxHeap has ignored or misused the channel and becayse tying the channel to SetMaxHeap may limit future evolution that could need the same type of notification for different reasons.

[rsc]

@aclements, what is the state of this proposal?
Is it the same as the current SetMaxHeap implementation sketch?
Something different?
And what is the relationship between this issue and #23044 and #16843?
Thanks.

[changkun]

Returning a channel is ambiguous to its caller. For instance: what happens if I close the channel? This requires detailed documentation.

More general thoughts about this proposal are, the userspace needs a runtime signal notifier and luckily we already have a practice in os.Signal. Here is a more general API that easy to scale to support more runtime signals:

package runtime

// Signal presents a runtime signal.
type Signal interface {
    String() string
    Signal() // to distinguish from other Stringers
}

// All kinds of signals from runtime.
var (
    GCBackPressure    Signal = ...
    GCCycleStart      Signal = ...
    GCCycleEnd        Signal = ...
    SchedBackPressure Signal = ...
)

func Notify(c chan <- Signal, sig ...Signal)

In userspace, we have what's similar to os.Signal:

package main

import "runtime"

func main() {
    c := make(chan runtime.Signal, 2)
    runtime.Notify(c, runtime.GCBackPressure, runtime.GCEnd)

    go func() {
        for {
            sig := <-c
            switch sig {
            case runtime.GCBackpressure:
                println("runtime sends a GC back pressure signal")
            case runtime.GCEnd:
                println("runtime sends a GC end signal")
            }
        }
    }()

    
    ... // do all kinds of application stuff
}

Thoughts?