Some proxied POST requests fail or hang when upstream does not consume body

[uhthomas]

1. What version of Caddy are you running (caddy -version)?

Caddy (untracked dev build)

2. What are you trying to do?

Upload files to a reverse proxy

3. What is your entire Caddyfile?

site.com {
  proxy / 127.0.0.1:1333 {
    max_fails 0
  }
}

4. How did you run Caddy (give the full command and describe the execution environment)?

$ sudo caddy -email="user@host.com" -agree

5. What did you expect to see?

Proxied requests and responses

6. What did you see instead (give full error messages and/or log)?

Requests were either not sent to the proxy server and responses were only partially sent (just the headers). Usually requests either end up hanging on the client side and never reach the proxy or the responses return the headers with a protocol exception (net::ERR_SPDY_PROTOCOL_ERROR)

This occurs when the proxy returns a status code outside the range of 200-399 despite having max_fails 0 set to prevent the backend from being marked as down. Even if this were the case, then this would not be expected behavior

No error messages are reported by Caddy when adding -log stdout to the command or anywhere else.

Code to reproduce (use the network inspector to view details)

[mholt]

Hi, thanks for the report.

Can you please build Caddy with the build.bash script and then re-run the -version flag? We need to know precise build info. Please ensure it is the same one with which you're encountering the problem.

Also, there is not enough details here to reproduce the error. Can you please provide the upstream server's source code so that we can run it and see the same results? As simple as possible would be good.

(Nvm, I see the link now.)

[uhthomas]

Sure, the output for $ caddy -version is Caddy 0.8.2 (+3c578df Fri Apr 22 21:58:13 UTC 2016)

[mholt]

Thanks for the build information!

Okay, so the JS has an error on line 20: Uncaught ReferenceError: res is not defined -- when I changed it to req the error went away.

Your upstream server is sending a 413 to all POST requests which is why you're getting the 413 error.

[uhthomas]

Sorry, my mistake! I guess I didn't catch it because requests never finished. The issue isn't about it returning 413s, but rather not returning anything at all, or sending anything to the upstream server for that matter. I have no idea what's causing it.

[mholt]

Hmm, just a note for anyone who looks to work on this: it seems that this can only be reproduced on an HTTPS connection as I have tried HTTP and it works fine.

[mholt]

This seems to have to do with the fact that it's a multipart-form request, for some reason. POSTing an empty request works fine. @nemothekid have you experienced this before? I can't quite figure out what's going on.

[nemosupremo]

I was able to reproduce this pretty reliably without going https (in FireFox).

(Stream of thought here...)
Adding this to the backend server "fixes" the issue.

diff --git a/c.go b/b.go
index 417aea3..bd1d613 100644
--- a/c.go
+++ b/b.go
@@ -10,6 +10,7 @@ import (
 func handler(w http.ResponseWriter, r *http.Request) {
        if r.Method == "POST" {
                log.Println("Received post request")
+               io.Copy(ioutil.Discard, r.Body)
                http.Error(w, http.StatusText(http.StatusRequestEntityTooLarge), http.StatusRequestEntityTooLarge)
                return
        }

Reading the entire body, before closing the request causes everything to go smoothly. This isn't a "fix" because the upstream server should be able to close the connection anytime it likes, regardless of the fact that Caddy hasn't finished sending the request to it. It looks like Caddy doesn't like that fact that the connection is being closed before all the data is sent and is hanging.

My guess is (didn't debug to find out) https://github.com/mholt/caddy/blob/master/middleware/proxy/reverseproxy.go#L205 returns a ErrConnectionClosed or something like that, and we see that error and decide to mark the backend as down.

Tangentially, I was checking out the reverseproxy.go in golang (where Caddy's reverseproxy.go is copied from) and saw they are using a CloseNotifier. I was wondering why that is, but it seems like this issue is the case.

Looking into requestCanceler, the docs say transport.CancelRequest is deprecated as it can't be used on http/2 connection so I'm not sure how useful copying the code, as is, is. Secondly I'm not sure is CancelRequest will give us an error on RoundTrip @ https://github.com/mholt/caddy/blob/master/middleware/proxy/reverseproxy.go#L205 leading us back to square one.

tl;dr reverseproxy.go has to detect that the backend server has decided to close the connection and decide how to communicate that back to the client.

[nemosupremo]

My hypothesis on copying the "requestcanceler" code didn't work.

[uhthomas]

Changing the round tripper to a http client straight up worked fine for me. Not sure if there are any disadvantages to this though.

Scratch that. I need to test my solutions more thoroughly before committing.

[uhthomas]

Looks as if golang/go#14061 might be related.

Derived from https://groups.google.com/forum/#!topic/golang-nuts/ZVfwwWttVYI

[mholt]

@nemothekid Very helpful -- thanks for looking into this! I also was wondering if the upstream was misbehaving but as you said, that shouldn't matter. Hmm, this is tricky!

Update: Ignore pretty much this whole post, I made a mistake in my testing (below).

Ah, and it's interesting that Firefox has a slightly different behavior. Still sporadic, but definitely different from Chrome, as at least one connection seems to succeed regularly:

Indeed some connections still hang.

Update: While writing this post, I am no longer able to reproduce the behavior locally. I'm serving over a test domain over HTTPS with a real certificate, hosted on my local computer. I'm accessing the endpoint exposed by Caddy, added some extra log.Print lines to see more of what is going on, and even tried messing with the payload: first removing the body entirely, then adding the multipart stuff, then re-enabling the content (the 5 MB of zeroes).

All I did was step away for a few hours, came back (both the servers were running the whole time, but I closed the computer), and it was working. I then restarted the servers and things are still working. Firefox and Chrome both show the % values going up until the upload is complete, after which they show a 413 error as expected.

The only changes I made were adding a few log lines. Strange, some?

[mholt]

Oh wait.

@nemothekid It's working for me now because, in adding logging to the upstream instance to verify how much of the response is being read, I'm consuming the response body with ioutil.ReadAll, which is the same as the workaround you suggested. * facepalm *

I will disable that logging for future debugging.

So far, your conclusion still seems correct: the upstream needs to be allowed to close the connection intentionally without Caddy marking the upstream as down.

[mholt]

I was running my tests without setting max_fails 0 like in @6f7262's instructions. When I did use that, I have discovered by outputting the outreq that the value of the X-Forwarded-For header very quickly becomes:

"X-Forwarded-For":[]string{"66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>..", 66.<my IP>.."

times like a bajillion.

It essentially tries requests infinitely for a minute, as Caddy is programmed to do when it's told to never assume a backend goes down. And each time it adds my IP address again to the header.

That's gotta increase memory usage. Whoops.

The errors that are being reported from reverseproxy.go are the likes of: Failed: stream error: stream ID 5; CANCEL

^ This behavior of multiplying that header value only happens sometimes; I may have to refresh the page to trigger it; other times, it just runs the requests once.

[nemosupremo]

(I don't know why it fails under Firefox, http/1.1)

AFAICT, It looks like a data race. I'm checking out a simpler case where var reqs = 1; in the JS code as that doesn't properly work either and is less complex than trying to debug 3 requests at once.

When var reqs = 1, the status code (413) is retrieved, but the body (which should be "Request Entity Too Large") never seems to be sent to the browser.

[uhthomas]

Speaking of things happening a bajillion times, I noted that when logging the error return from RoundTrip, it would be stream error: stream ID 5; CANCEL over and over, not sure if this is related.