Allow the encoder to split headers across frames

[Tratcher]

The HPackEncoder has a potential infinite loop today.
https://github.com/aspnet/KestrelHttpServer/blob/55e5e564226e7b27742c91f46ea71841071e4ac3/src/Kestrel.Core/Internal/Http2/HPack/HPackEncoder.cs#L35-L42

EncodeHeader returns false if it does not have enough space to encode the value. If this header was larger than max frame size then no data will be available for the frame at all and length will be 0. The server will be stuck in an infinite loop sending zero length continuation frames.

aspnet/KestrelHttpServer#2893 added an exception to break the infinate loop, but it did so by throwing and very ungracefully aborting the socket.

Proposal: The spec allows splitting encoded headers across frames. This would allow the encoder to pack frames more densely and always make progress.

Priority: Not urgent, the response header would need to be larger than the max frame size (16kb). The client and server can also raise that frame size. Implementing real response header compression would allow even larger values before hitting the frame size limit.

[gfr-g]

Hello, I would like to take a look at the issue.

Context

Header lists are collections of zero or more header fields. When transmitted over a connection, a header list is serialized into a header block using HTTP header compression [COMPRESSION]. The serialized header block is then divided into one or more octet sequences, called header block fragments, and transmitted within the payload of HEADERS (Section 6.2), PUSH_PROMISE (Section 6.6), or CONTINUATION (Section 6.10) frames.

Brief recap

As already stated by @Tratcher, if an header doesn't completely fit inside the payload of the current frame, it will be written in the next continuation frame, leaving part of the current frame payload "empty" (or triggering the bug in case the header itself is bigger than the whole payload of a frame).
The HPackEncoder use a bool semantic for its methods to communicate with the caller whether or not it was able to write a given header inside the current frame payload.

Speculation on possible solutions

1.

• change the "encoding methods" so that they keep writing until the payload of the current frame is full

• change the return types from bool to something richer that can carry two pieces of information: the state of the encoding (header (or headers) fully or partially written) and eventually the encoded bytes remaining from a partial write, so that the Http2FrameWriter can initialize the payload correctly on the next continuation frame.

Impact
I believe that at least the HPackEncoder, the HPackHeaderWriter and the Http2FrameWriter have to be changed. The memory footprint should increase in case of a partial write because the encoded bytes that didn't fit in the current frame have to be saved and copied back into the continuation frame payload.

2.

Another solution would be to change the current design and use something like an observer pattern where the HPackHeaderWriter is the observable and the HTTP2FrameWriter is the observer. The HPackHeaderWriter encodes and writes the headers until it reaches the end of the payload. At that point it stops encoding and signals the HttP2FrameWriter so that it can send the current frame. When this finishes, the encode resume writing on the continuation frame payload from the beginning.

Impact
Code changes should be bigger but it should solve the problem of copying the encoded bytes that didn't fit in the current frame back and forth but it would add some overhead for having an actual instance of the HPackHeaderWriter (right now is static) per connection, regardless of whether or not the headers fully fit in one frame (which is, I believe, the most common case given a 16kb payload).

There might be way better ways to solve the issue, I'm looking forward to hear from the team, if the team still see value in improving this part.

Thank you.

[Tratcher]

As you noticed, this issue is pretty complicated for a fairly minor encoding optimization. You're welcome to explore it and see if it can be done without being too disruptive/complex/slow. You may also want to wait another week or so because there's some ongoing refactoring in this area for #30235.