H2C Server causes Truncated Body On HTTP2 Requests with a Reverse Proxy, but only when writing a new response body. (Server Reset Stream)

[austincollinpena]

What version of Go are you using (1.153)?

$ 1.153

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Output


$ go env

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Austin Pena\AppData\Local\go-build
set GOENV=C:\Users\Austin Pena\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Austin Pena\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\AUSTIN~1\AppData\Local\Temp\go-build661771174=/tmp/go-build -gno-record-gcc-switches

What did you do?

I was getting timeouts on Go Playground otherwise I would have posted it there.

Here's the code for reproducibility. Please try to access https://easy-dp.ngrok.io to see the issue.

1. Create a Reverse Proxy accessing Gzipped/ Br encoded Content
2. Request a publicly available URL, I just grabbed Google Analytics
3. Attempt to encode and decode the response via an http2 connection with a proxy.modifyresponse function
4. Watch as content is dropped.

However, this only occurs under the following conditions:

• Under SSL, like with https://easy-dp.ngrok.io
• When running a proxy.ModifyResponse function
• Decompressing and re-compressing the body (for example, just reading and rewriting the body to new bytes works)

package main

import (
	"bytes"
	"compress/gzip"
	"fmt"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
	"io/ioutil"
	"net/http"
	"net/http/httputil"
	"strconv"
	"time"
)

func ForwardAnalytics(req *http.Request) {
	req.URL.Scheme = "https"
	req.URL.Host = "www.google-analytics.com"
	req.Host = "www.google-analytics.com"
	req.URL.Path = "/analytics.js"
	req.Header.Set("Accept-Encoding", "gzip")
}

func ModifyAnalytics(r *http.Response) error {
	bytesFromBody, err := ioutil.ReadAll(r.Body)
	defer r.Body.Close()
	if err != nil {
		return nil
	}
	if r.Header.Get("Content-Encoding") == "gzip" {
		gzipReader, err := gzip.NewReader(bytes.NewBuffer(bytesFromBody))
		if err != nil {
			return nil
		}
		defer gzipReader.Close()
		readableBytes, err := ioutil.ReadAll(gzipReader)
		var b bytes.Buffer
		gzipWriter, err := gzip.NewWriterLevel(&b, gzip.DefaultCompression)
		if err != nil {
			return nil
		}
		defer gzipWriter.Close()
		writtenLen, err := gzipWriter.Write(readableBytes)
		fmt.Println("Wrote ", writtenLen)
		if err != nil {
			return nil
		}
		r.ContentLength = int64(len(readableBytes))
		r.Header.Set("Content-Length", strconv.FormatInt(int64(len(readableBytes)), 10))
		r.Body = ioutil.NopCloser(&b)

		return nil
	} else {
		return nil
	}
}


func handleProxy(w http.ResponseWriter, req *http.Request) {
	proxy := httputil.ReverseProxy{
		Director: ForwardAnalytics
	}
	proxy.ModifyResponse = ModifyAnalytics
	proxy.ServeHTTP(w, req)

}

func main() {
	h2s := &http2.Server{
		IdleTimeout: 20 * time.Second,
	}
	mux := http.NewServeMux()
	mux.HandleFunc( "/", handleProxy)
	s := &http.Server{
		ReadHeaderTimeout: 20 * time.Second,
		ReadTimeout:       10 * time.Second,
		WriteTimeout:      30 * time.Second,
		Addr:              "localhost:8456",
		Handler:           h2c.NewHandler(mux, h2s),
	}
	s.ListenAndServe()
}

What did you expect to see?

I expect to see the ability to open the bytes, modify them, and update the response body on an H2C connection

What did you see instead?

Two things of note happen:

1. Chrome gives a nice little error that expands upon what's going on

{"params":{"description":"Server reset stream.","net_error":"ERR_HTTP2_PROTOCOL_ERROR","stream_id":5},"phase":0,"source":{"id":1493828,"start_time":"732370299","type":1},"time":"732375561","type":224},

2. Under the normal http connection, there's no problem, but under the https connection the script may or may not print out to a certain length. Sometimes it doesn't print at all, sometimes it prints about 30%.

This is a cross browser issue.

[seankhliao]

The Content-Length header indicates the size of the entity body in the message, in bytes. The size includes any content encodings (the Content-Length of a gzip-compressed text file will be the compressed size, not the original size).

[austincollinpena]

@seankhliao Thanks for taking the time to point that out. I had originally done that, but failed to close the gzip writer correctly. I was having this issue until I rewrote my code to the following:

	if r.Header.Get("Content-Encoding") == "gzip" {
		gzipReader, err := gzip.NewReader(bytes.NewBuffer(bytesFromBody))
		if err != nil {
			return nil
		}
		defer gzipReader.Close()
		readableBytes, err := ioutil.ReadAll(gzipReader)
		var b bytes.Buffer
		gzipWriter, err := gzip.NewWriterLevel(&b, gzip.DefaultCompression)
		if err != nil {
			return nil
		}
		writtenLen, err := gzipWriter.Write(readableBytes)
		gzipWriter.Close() // This was the culprit. It needed to be closed here
		fmt.Println("Wrote ", writtenLen)
		if err != nil {
			return nil
		}
		r.ContentLength = int64(b.Len())
		r.Header.Set("Content-Length", strconv.FormatInt(int64(b.Len()), 10))
		r.Body = ioutil.NopCloser(&b)
		return nil
	}

It's interesting to me that it only happened over an H2C connection though. Perhaps it was just due to latency? Hard to say.