transport: reuse memory on read path

[MakMukhi]

One possible idea is to use a leaky buffer.

[dfawley]

Specific areas where we would like to do this:

• Codec output buffer (requires codec API extension)
• Reading from transport (I believe this is Reuse receive buffer #869)

[muirdm]

Is someone actively working on this? I've been playing around with different buffer pool strategies, but don't want to look too much more if I'm just duplicating work.

I've been trying bucketed []byte buffer pool where there are tiers of buffer sizes. For example, if you ask for a 700 byte buffer, it might pull a buffer from the 1024 byte pool, slice it down to 700 bytes and return it.

I'm trying a *sync.Pool implementation (one *sync.Pool per buffer size bucket) and a leaky buffer implementation (one buffered chan []byte per buffer size bucket). They certainly help, but neither one has plucked all the low-hanging garbage fruit on the first try.

[awalterschulze]

I would prefer that the codec object itself is in a sync pool and is assumed to be unsafe to use concurrently. This way the codec itself can optimize its buffer use. This codec that I would like to use with grpc is the one that I have implemented in github.com/gogo/protobuf/codec

…

On Sat, 2 Dec 2017, 01:18 Muir Manders, ***@***.***> wrote: Is someone actively working on this? I've been playing around with different buffer pool strategies, but don't want to look too much more if I'm just duplicating work. I've been trying bucketed []byte buffer pool where there are tiers of buffer sizes. For example, if you ask for a 700 byte buffer, it might pull a buffer from the 1024 byte pool, slice it down to 700 bytes and return it. I'm trying a *sync.Pool implementation (one *sync.Pool per buffer size bucket) and a leaky buffer implementation (one buffered chan []byte per buffer size bucket). They certainly help, but neither one has plucked all the low-hanging garbage fruit on the first try. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1455 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvsLVtUmZ3k_U6PNTHhMQgt9P0qZ57qks5s8JddgaJpZM4O_JA3> .

[awalterschulze]

Here is the correct link https://github.com/gogo/protobuf/blob/master/codec/codec.go

…

On Sat, 2 Dec 2017, 10:42 Walter Schulze, ***@***.***> wrote: I would prefer that the codec object itself is in a sync pool and is assumed to be unsafe to use concurrently. This way the codec itself can optimize its buffer use. This codec that I would like to use with grpc is the one that I have implemented in github.com/gogo/protobuf/codec On Sat, 2 Dec 2017, 01:18 Muir Manders, ***@***.***> wrote: > Is someone actively working on this? I've been playing around with > different buffer pool strategies, but don't want to look too much more if > I'm just duplicating work. > > I've been trying bucketed []byte buffer pool where there are tiers of > buffer sizes. For example, if you ask for a 700 byte buffer, it might pull > a buffer from the 1024 byte pool, slice it down to 700 bytes and return it. > > I'm trying a *sync.Pool implementation (one *sync.Pool per buffer size > bucket) and a leaky buffer implementation (one buffered chan []byte per > buffer size bucket). They certainly help, but neither one has plucked all > the low-hanging garbage fruit on the first try. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#1455 (comment)>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ABvsLVtUmZ3k_U6PNTHhMQgt9P0qZ57qks5s8JddgaJpZM4O_JA3> > . >

[muirdm]

There is notable garbage produced outside of proto marshaling in the http2 transport, so I was hoping to address that too. I was looking at more generic []byte buffer reuse since it could be used by the existing codec, and used by the http2 transport.

How is pooling the codec better than just pooling the []byte slice? The existing codec could pull its []byte from []byte pool, and also implement similar gogoproto optimized logic taking advantage of Size() and MarshalTo() as appropriate.

[awalterschulze]

Thats interesting. I didn't think that the http2 transport could also benefit from buffer pooling. At the previous company I worked, we had a similar buffer pooling scheme for a different purpose. The API asks for a buffer size and returns that size of byte buffer for you. It does this by looking through its slice of buffers for a big enough buffer and then returning the resized to your smaller size. When buffers are returned they are stretched back to their original capacity.
This buffer pool also had retention policies, which were a bit more complex, but implemented orthogonally with the buffer pool.

[dfawley]

My thinking for the codec was to add an optional method to it that, if implemented, would be called when the buffer it returns from Marshal is no longer needed by grpc. Then the codec could re-use that memory. Without something like this, it's impossible for the codec to safely reuse old memory buffers.

The http2 transport uses byte buffers to hold data it reads from the network; this can and should be re-used with a pool as well.

[muirdm]

My thought was instead of returning the buffer to the codec, return the buffer to the generic buffer pool instead. Then anyone can pull a buffer from the pool, and consumers of the buffers can put them back in the pool when they are done. It is safe to never return a buffer, or even to return a buffer that you did not get from the pool originally. On the other hand it is very unsafe to return a buffer and then still use it, or a slice of it. We would need some approach to make ourselves confident that use-after-release doesn't happen.

[awalterschulze]

What are other languages doing to optimize this situation? Maybe we can learn something from c++ or even Java.

…

On Sat, 2 Dec 2017, 23:26 Muir Manders, ***@***.***> wrote: My thought was instead of returning the buffer to the codec, return the buffer to the generic buffer pool instead. Then anyone can pull a buffer from the pool, and consumers of the buffers can put them back in the pool when they are done. It is safe to never return a buffer, or even to return a buffer that you did not get from the pool originally. On the other hand it is very unsafe to return a buffer and then still use it, or a slice of it. We would need some approach to make ourselves confident that use-after-release doesn't happen. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1455 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvsLa5exKkGKa5OEtS5Rb-3p0UBGuf_ks5s8c6GgaJpZM4O_JA3> .