RequestURI path is modified when path has multiple `/` characters

[youngm]

We are running cf-release 183.

It appears that the gorouter is modifying the Request Path of a request if prefixed with multiple '/' characters.

Example Request: http://test.app.com/bob
Router Log in Loggregator: GET /bob HTTP/1.1" 404 15

Example Request: http://test.app.com//bob (node the //)
Router Log in Loggregator: GET http://bob HTTP/1.1

Note the addition of "http:" to the request path. This is what I'd expect to see:
Router Log in Loggregator: GET //bob HTTP/1.1

I would expect the go router to pass this path to the server unchanged.

You can see similar behaviour on pws. Though something appears to be stripping one of the slashes as you only get an error with 3 ///.

Example pws Request: http://boshartifacts.cloudfoundry.org///joe
WEBrick response: Bad Request bad URIhttp://joe'.`

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. You can view the current status of your issue at: https://www.pivotaltracker.com/story/show/80170710.

[youngm]

I think this test duplicates the issue in proxy_test.go:

    It("maintains double /", func() {
        shouldEcho("//abc", "//abc")
    })

Returns a 400 Status Code instead of 200 from the proxy server.

[fraenkel]

According to rfc3986, an absolute path begins with / but must not '//'. The 400 is a valid response to a request the is invalid.
I wouldn't be surprised if many runtimes are being loose in their definition/parsing of a request path.
If we augment your test to something like /a/////b, you will see that it passes.

The failure occurs when the path is parsed as a valid url.ParseRequestURI inside request.go

[youngm]

Thanks for responding @fraenkel I agree that // is an invalid Request. My test was wrong. But don't you think the router should pass it through unchanged anyway instead of munging it into http://?

Perhaps I'll have to muddle my way through creating a better unit test to show that the url is getting munged.

[youngm]

@fraenkel This is hopefully a better test that shows the router is modifying paths that start with //. Even though // is an invalid path I don't think the router should mess with it.

    It("maintains a double slash at beginning of path", func() {
        ln := registerHandler(r, "test", func(x *test_util.HttpConn) {
            x.CheckLine("GET //test_path HTTP/1.1")
        })
        defer ln.Close()

        x := dialProxy(proxyServer)

        x.WriteLines([]string{
            "GET //test_path HTTP/1.1",
            "Host: test",
        })

        x.ReadResponse()
    })

[fraenkel]

For kicks, I tried a few other servers to see what happens with your //test_path request.
What I got back were:

404 Not Found: Requested route ('test_path') does not exist.
and
404 The requested URL /test_path

// is not valid period, and there are many translations to make it valid all of which change what is there. Gorouter is actually not doing anything but the underlying golang http server is doing all the munging that is being passed through. If any change is to be made its in the core golang code.

[youngm]

@fraenkel Hmm....can you confirm you are seeing the same test results as me?

When I run that test I get the following failure:

  Expected
      <string>: GET http://test_path HTTP/1.1
  to equal
      <string>: GET //test_path HTTP/1.1

You think that it is correct that the path is changing from //test_path to http://test_path? Or do you think the backend server is changing that path? That path change shows up in loggregator logs so I don't think it is the back end server changing the path then somehow returning it back to the router to be logged?

[youngm]

@fraenkel I re-read your comment: Gorouter is actually not doing anything but the underlying golang http server is doing all the munging that is being passed through. If the munging you are referring to is the "http://" munging I can acknowledge that it could be a core golang code bug. But, I submit it here because I believe it is invalid functionality exibited in the gorouter component regardless of if the fault lies with golang code or gorouter code. // is an invalid path. I get that. But I don't think it is valid behaviour for a reverse proxy to be changing requests invalid or not when it could just forward it on.

Let me give you background on this issue. We are trying to deploy a thirdparty commercial product to Cloud Foundry. This commercial product has a compiled thick client with a "bug" in it that sends all it's rest requests with //. It works great if they hit a dev server directly but fails if they deploy their app to cloud foundry. So, this issue isn't simply a matter of someone fat fingering a url in their browser. We cannot change the code that is adding //. It works when they hit their server directly. Fails when going through a CF router. From their perspective CF is not acting as a transparent platform since it is munging the requests sent to their server.

Hopefully that background helps.

[youngm]

Here is another test case going against the loggregator log produced by the router. I don't see how the back end server could be changing the url the router is logging, right?

    It("Logs a doubleslash", func() {
        ln := registerHandler(r, "test", func(x *test_util.HttpConn) {
            x.ReadRequest()
            x.WriteLines([]string{
                "HTTP/1.1 200 OK",
                "Content-Length: 0",
            })
        })
        defer ln.Close()

        x := dialProxy(proxyServer)

        x.WriteLines([]string{
            "GET //test_path HTTP/1.0",
            "Host: test",
        })

        x.CheckLine("HTTP/1.0 200 OK")

        var payload []byte
        Eventually(func() int {
            accessLogFile.Read(&payload)
            return len(payload)
        }).ShouldNot(BeZero())
        Ω(string(payload)).To(MatchRegexp("GET //test_path"))
    })

To be clear. This is the output I get when this test fails:

  Expected
      <string>: test - [08/10/2014:10:03:47 -0600] "GET http://test_path HTTP/1.0" 200 0 "-" "-" 127.0.0.1:43082 x_forwarded_for:"127.0.0.1" vcap_request_id:78486ac8-b3e7-4aad-4e3e-c95f994b9df9 response_time:0.007782678 app_id:

  to match regular expression
      <string>: GET //test_path

[dieucao]

@youngm We would be fine to take a PR on this if you wouldn't mind working on it.

[youngm]

@dieucao This is one of those hard ones I was hoping you guys could do. :) But, I can see it isn't a priority for you so I'll take a look. Hopefully it doesn't involve rewriting all the golang http server libraries.

Thanks,
Mike

[youngm]

Well, I found the problem code: http://golang.org/src/pkg/net/url/url.go#692

In all other ways it appears that the Opaque variable is truly Opaque in every way except in this one way. And I can see no way to supply a truly Opaque path to request.go without completely rewriting the request package. Why is URL not an interface? I hate go lang. :)

[jfmyers9]

Hi @youngm,

We agree that the gorouter should not modify requests, however we feel uncomfortable putting a work around in the gorouter for a bug from golang core. If you wish to pursue this further, we suggest that you open up an issue with golang.

Thanks.

@jfmyers9 && @sergueif, CF Runtime Team

[youngm]

Sounds fair. I looked into it and it is not worth the effort for me to lobby golang for a fix.

• I don't have a relationship with the golang community.
• This issue is really only an issue when using golang's http package as a transparent proxy probably not the original purpose of the http package. Not something I feel like putting effort into debating in the golang community.

You can close the issue if you wish. At the very least perhaps you can keep this issue in mind next time the router is up for a rewrite.

Mike

[zrob]

We have no plans to resolve this issue at this point.

We will leave the issue open as a KB for others running into this issue.