x/net/http2: TestContentEncodingNoSniffing failures with "connection reset by peer"

[bcmills]

2021-06-14T18:27:18-04defd4/linux-ppc64le-buildlet
2021-04-05T18:03:19-a5a99cb/linux-ppc64-buildlet
2021-03-31T06:09:03-cb1fcc7/linux-ppc64le-power9osu
2021-03-29T18:18:59-df645c7/linux-ppc64le-buildlet

CC @neild @tombergan @laboger

It's not clear to me whether this is a flaky Go test or a platform bug.

[bcmills]

Turns out not to be specific to PPC64, so this is looking like a flaky test or x/net/http2 bug rather than a platform bug.

--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: Failed to fetch URL: write tcp 127.0.0.1:56032->127.0.0.1:34169: write: connection reset by peer
FAIL
FAIL	golang.org/x/net/http2	18.129s

greplogs --dashboard -md -l -e 'FAIL: TestContentEncodingNoSniffing .*\n( .*\n)* .*: connection reset by peer' --since=2021-06-16

2022-01-21T00:39:55-0dd24b2-32636cd/linux-amd64-clang
2021-11-03T00:07:38-4a448f8-49a0063/linux-386-clang
2021-09-03T01:38:54-8d99171-9f69a44/linux-arm64-aws

[bcmills]

These failures do not appear to be specific to a single subtest.

[bcmills]

The subtests of this test share a single Transport, but tear down and set up a new serverTester for each subtest.

@neild, could that cause this failure mode if two of the serverTester instances happen to pick the same port?

[bcmills]

(Nope, that appears not to be the problem: setting -httptest.serve to a specific port does not cause the test to fail.)

[bcmills]

Setting -httptest.serve to explicitly reuse the same port seems to increase the failure rate substantially.

~/x/net$ go test ./http2 -run=TestContentEncodingNoSniffing -httptest.serve=127.0.0.1:0 -count=10000 -timeout=1h
ok      golang.org/x/net/http2  429.798s

~/x/net$ go test ./http2 -run=TestContentEncodingNoSniffing -httptest.serve=127.0.0.1:8080 -count=10000 -timeout=1h
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:55808->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:55810->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56964->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40744->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40750->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46014->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46036->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46090->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56912->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35440->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40690->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40692->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45978->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45982->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.02s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45990->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45992->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45996->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46002->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46010->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46014->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46022->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46028->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:47740->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35336->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35340->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35342->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35348->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45878->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45908->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45914->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45918->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:47706->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.06s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56668->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.02s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56672->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35158->127.0.0.1:8080: write: connection reset by peer
FAIL
FAIL    golang.org/x/net/http2  430.663s
FAIL

[gopherbot]

Change https://golang.org/cl/380154 mentions this issue: http2: in TestContentEncodingNoSniffing, do not allow the Transport to outlive the serverTester

[bcmills]

Marking as release-blocker: this test empirically affects multiple first-class ports, and I've mailed what I believe to be a fix.