net/http: provide way to get Request write error and HTTP response?

[tarm]

(As Brad mentioned on go-nuts, this is likely a race between two different responses and may not be a bug. Nonetheless, here is a report to track it if you want to consider changing the behavior in the future.)

On linux x64, the following program will go test -v with "Failed 0 out of 100" with go1.4. In go1.5beta1, I often get results like

...
http_test.go:33: 98 Put http://localhost:8080/: write tcp 127.0.0.1:50310->127.0.0.1:8080: write: connection reset by peer
http_test.go:38: Failed 68 out of 100

This seems to be caused by commit 1045351 and reverting that commit makes the test pass again.

package main

import (
        "net/http"
        "strings"
        "testing"
)

const contentLengthLimit = 1024 * 1024 // 1MB                                                                                                                             

func handler(w http.ResponseWriter, r *http.Request) {
        if r.ContentLength >= contentLengthLimit {
                w.WriteHeader(http.StatusBadRequest)
                r.Body.Close()
                return
        }
        w.WriteHeader(http.StatusOK)
}

func TestHandler(t *testing.T) {
        http.HandleFunc("/", handler)
        go http.ListenAndServe(":8080", nil)

        fail := 0
        count := 100
        for i := 0; i < count; i++ {
                r := strings.NewReader(strings.Repeat("a", 10000000))
                req, err := http.NewRequest("PUT", "http://localhost:8080/", r)
                client := &http.Client{}
                resp, err := client.Do(req)
                if err != nil {
                        fail++
                        t.Log(i, err)
                        //t.Fatal(err)                                                                                                                                    
                } else if resp.StatusCode != 400 {
                        t.Fatalf("Expected the status 400, got %v instead.", resp.StatusCode)
                }
        }
        t.Logf("Failed %v out of %v\n", fail, count)
}

[mikioh]

@tarm,

See http://golang.org/pkg/net/http/#Client.Do. I think both versions behave identically when you call resp.Body.Close.

[tarm]

@mikioh I agree that the client behavior did not change. I think the server changed to call resp.Body.Close() sooner, which changed the result returned by the client in my tests. That may not be a bug.

Specifically, the client documentation makes clear that When err is nil, resp always contains a non-nil resp.Body but it leaves open whether err and resp could both be non-null at the same time. http://golang.org/pkg/io/#Reader is an example were both data and err may be returned from the same call.

In a full duplex connection (like net.Conn or io.ReadWriter), the client can Read() a complete and valid response even as the Write() fails. What should the http.Client return in that case?

Perhaps the title should be "net/http: should http.Client return a non-nil err and resp at the same time?"

[bradfitz]

@rsc, I'm wondering if we should do anything to address this.

In summary: the http.RoundTripper interface says you get either a *Response, or an error.

But in the case of a client writing a large request and the server replying prematurely (e.g. 403 Forbidden) and closing the connection without reading the request body, what does the client want? The 403 response, or the error that the body couldn't be copied?

In addressing a server DoS issue, Go 1.5 now more aggressively closes connections instead of reading forever when the Handler doesn't read to EOF.

There are multiple potential fixes, none a complete solution:

a) change the http.RoundTripper interface to say it can return (non-nil, non-nil). This is problematic for compatibility. Old code wouldn't know to potentially close the non-nil *Response.Body when err is already non-nil.

b) make the Server delay the TCP shutdown for N milliseconds after writing the response, thereby increasing the chance that the client sees our response. We do this elsewhere.

c) make the Transport give an N millisecond (optional, tunable on *Transport?) advantage to responses over body write errors. e.g.:

$ git di
diff --git a/src/net/http/transport.go b/src/net/http/transport.go
index 41fc6d0..6d4b50a 100644
--- a/src/net/http/transport.go
+++ b/src/net/http/transport.go
@@ -1165,6 +1165,10 @@ WaitResponse:
                case err := <-writeErrCh:
                        if err != nil {
                                re = responseAndError{nil, err}
+                               select {
+                               case re = <-resc:
+                               case <-time.After(20 * time.Millisecond):
+                               }
                                pc.close()
                                break WaitResponse
                        }

We could do (c) easily and probably even (b) for Go 1.5 if this is a concern. It's very late, though. And we already had this race/ambiguity in all previous Go releases, so maybe it's not a big deal. It's just changed slightly now.

I'd like to think of a better story for Go 1.6 but I don't have any great answers yet.

[nightlyone]

What about a special error type when implementing a) ?
So a nil response and a type HTTPError struct { Code int; Err error } implementing the error interface can be returned.

Such a type would be extremely useful in a lot of other situations too. So it is very useful to provide it anyway.

[rsc]

Doing (c) sounds great for Go 1.5. If (b) is as simple as your patch for (c), we can do (b) too. I agree that doing (a) is basically a non-starter, since it will break all the old code that assumes either-or.

Introducing new API like an HTTPError as in @nightlyone's comment is not something we should do now. It also has serious backwards compatibility issues, since people have learned that HTTP error codes come back as ordinary responses.

[rsc]

Brad, is #11020 the same problem?

[bradfitz]

No, #11020 is different but easy.

[gopherbot]

CL https://golang.org/cl/12590 mentions this issue.

[gopherbot]

CL https://golang.org/cl/12658 mentions this issue.

[tarm]

@bradfitz are you planning further changes for this ticket? If not I think it can be closed.

I opened it because some of our unit tests started failing after the update to 1.5beta1. In addition to the fix c) that you put in, we also modified our unit tests now and this is not an issue for us and likely not for anyone else either.

[bradfitz]

@tarm I'll close or decide what to do during the Go 1.9 cycle.

[d33d33]

This issue seems to be back. I am running a program under go1.8.3 which encounter really often the following error: write tcp x.x.x.x:40322->x.x.x.x:443: write: broken pipe. This program perform http request to a server (Jetty) that closed the connection immediately after issuing a 403 error code while the client continue to send data.

The @bradfitz fix seems to have been override by 7fa9846 so I simply reapply the following commit 0c2d3f7. This do the trick, I no longer experience write: broken pipe :)
You can find the full diff here e3adabb

[StevenLeRoux]

Hi, I'm facing the same issue.

Currently as a workaround, we maintain our own Golang fork, but it's clearly not a long term solution.

It'd be great to see this fixed for 1.10.

[rsc]

Moving to Go 1.11, sorry.

[thinkhy]

Any update for this issue? I hit this one when run IPFS.

[bradfitz]

Nope. Updates would be posted here.

[jsarenik]

Greetings! go1.15.5 linux/amd64 and latest go-ipfs (ipfs/kubo@895b340e0).

Ref: ipfs/kubo#3855

Every time I run it, I get this:

$ ipfs add -Qr dictionary/ >/dev/null
Error: read tcp 127.0.0.1:59078->127.0.0.1:5001: use of closed network connection

The same was happening with the release go-ipfs 0.7.0 binary.

UPDATE:
Apologies for noise here. My error was caused by a leftover dictionary/sed6iJvHZ file which appears when a sed -i ... is killed. The file is created with umask 777 so with ls -l it starts with ---------. After removing the file everything works.

$ rm dictionary/sed6iJvHZ
rm: remove write-protected regular file 'dictionary/sedysQ2NJ'? y

[seankhliao]

Seems to no longer reproduce.