x/net/http2: return flow control when stream closes with buffered data

[wujieliulan]

I ran into a case where my http2 transport cannot send any data to my http2 server with the existing connection. The issue is client sent some data in a stream, when server receives it, there was a stream error (or stream closed) and the data received was discarded., server never got a chance to send windowupdate for this client connection, client can no longer send more data, even for new streams.

I did some simple changes and this issue is fixed, but I am not sure if this is the right way to fix. Could you advice? Is there a similar issue for server sending to client?

Here is the change I made:

diff --git a/http2/server.go b/http2/server.go
index f368738..b924c3d 100644
--- a/http2/server.go
+++ b/http2/server.go
@@ -1277,6 +1277,9 @@ func (sc *serverConn) processSettingInitialWindowSize(val uint32) error {

 func (sc *serverConn) processData(f *DataFrame) error {
        sc.serveG.check()
+       if len(f.Data()) > 0 {
+               sc.sendWindowUpdate(nil, len(f.Data())) // conn-level
+       }
        // "If a DATA frame is received whose stream is not in "open"
        // or "half closed (local)" state, the recipient MUST respond
        // with a stream error (Section 5.4.2) of type STREAM_CLOSED."
@@ -1764,7 +1767,7 @@ func (sc *serverConn) noteBodyReadFromHandler(st *stream, n int) {

 func (sc *serverConn) noteBodyRead(st *stream, n int) {
        sc.serveG.check()
-       sc.sendWindowUpdate(nil, n) // conn-level
+       //sc.sendWindowUpdate(nil, n) // conn-level
        if st.state != stateHalfClosedRemote && st.state != stateClosed {
                // Don't send this WINDOW_UPDATE if the stream is closed
                // remotely.

[bradfitz]

Your patch looks like you're just sending infinite flow control, giving them new tokens whenever they send new stuff, without any user acknowledgement.

Do you have a more complete repro or instructions?

[bradfitz]

Also, can you try Go 1.7? @adg changed some flow control stuff recently.

[wujieliulan]

The logic is this: when server receives a data frame, it should send WindowUpdate(len(data) ) for this client connection, even if the stream has error or closed and data can not be delivered to the handler. When data is read by handler, then it should send WindowUpdate for this stream.

I used the latest golang.org/x/net/http2.

[bradfitz]

The logic is this: when server receives a data frame, it should send WindowUpdate(len(data) ) for this client connection, even if the stream has error or closed and data can not be delivered to the handler.

But that doesn't look like what your patch does. You seem to be always sending flow control, not just for closed streams.

So would a better summary of the problem be: "the server doesn't refresh connection-level flow control when clients send DATA frames to closed streams" ?

[wujieliulan]

You summary is accurate. The patch does not always send flow control, it separates sending connection-level flow control and stream level:
when a data frame is received, send connection-level flow control of len(data)
when data is read by handler, send stream-level flow control of len(data)

That being said, I am not trying to propose a fix, just trying to provide more info to help find the issue.

[wujieliulan]

It seems transport has similar issue: the client (or transport) doesn't refresh connection-level flow control when server sends DATA frames to closed streams".

In transport.go, sending flow control for connection-level and stream-level are both in Read(). I think connection-level flow control should be sent whenever a data frame is received, stream-level flow control should be sent when data is read by handler.

[bradfitz]

I think connection-level flow control should be sent whenever a data frame is received, stream-level flow control should be sent when data is read by handler.

No, that doesn't sound right. If that were done, clients could repeatedly open new streams (which start with a default flow control) and write an unbounded amount.

I think we need to verify buffered flow control (from server-read but handler-unread bytes) is always returned on stream close, or when writing to a stream not accepting data.

I've been traveling the past 20 hours (to an HTTP workshop, as it happens) so I'll look at this after I've had some sleep and get a free moment.

If you or somebody else wants to write stand-alone repro unit tests in the x/net/http2 style, using its existing test framework, that would be helpful.

[bradfitz]

This is actually very easy to reproduce:

type funcReader func([]byte) (n int, err error)

func (f funcReader) Read(p []byte) (n int, err error) { return f(p) }

func TestUnreadFlowControlReturned(t *testing.T) {
        unblock := make(chan bool, 1)
        defer close(unblock)

        st := newServerTester(t, func(w http.ResponseWriter, r *http.Request) {
                <-unblock
        }, optOnlyServer)
        defer st.Close()

        tr := &Transport{TLSClientConfig: tlsConfigInsecure}
        defer tr.CloseIdleConnections()

        for i := 0; i < 10; i++ {
                println(i)
                body := io.MultiReader(
                        io.LimitReader(neverEnding('A'), 16<<10),
                        funcReader(func([]byte) (n int, err error) {
                                unblock <- true
                                return 0, io.EOF
                        }),
                )
                req, _ := http.NewRequest("POST", st.ts.URL, body)
                res, err := tr.RoundTrip(req)
                if err != nil {
                        t.Fatal(err)
                }
                res.Body.Close()
                println(i, "done")
        }

}

As feared, it hangs after 3 iterations, blocked on waiting for flow control to write the 4th body:

goroutine 37 [semacquire]:
sync.runtime_notifyListWait(0xc420014ed0, 0x0)
        /Users/bradfitz/go/src/runtime/sema.go:267 +0x122
sync.(*Cond).Wait(0xc420014ec0)
        /Users/bradfitz/go/src/sync/cond.go:57 +0x80
golang.org/x/net/http2.(*clientStream).awaitFlowControl(0xc4202ea280, 0x1, 0x0, 0x0, 0x0)
        /Users/bradfitz/src/golang.org/x/net/http2/transport.go:989 +0x11c
golang.org/x/net/http2.(*clientStream).writeRequestBody(0xc4202ea280, 0x507dc0, 0xc4202ec160, 0x12641e0, 0xc4202de150, 0x0, 0x0)
        /Users/bradfitz/src/golang.org/x/net/http2/transport.go:904 +0x23c
golang.org/x/net/http2.(*Transport).getBodyWriterState.func1()
        /Users/bradfitz/src/golang.org/x/net/http2/transport.go:1824 +0xa1
created by golang.org/x/net/http2.bodyWriterState.scheduleBodyWrite
        /Users/bradfitz/src/golang.org/x/net/http2/transport.go:1871 +0x8e

This is pretty bad and I'm surprised nobody has encountered this earlier. Thanks for reporting it.

@adg, @broady, this isn't a regression from Go 1.6, but it's pretty nasty and I think the fix will be easy.