Percent-decode more stuff?

[SimonSapin]

In #77, I was surprised that . was such a special case. It is indeed in Firefox 44, but in Chromium 48 other characters are also percent-decoded: -, _, ~, and ASCII alpha-numerics. (I don’t have other browsers to test right now.)

http://software.hixie.ch/utilities/js/live-dom-viewer/?saved=3878

It may be worth testing other URL components too.

[annevk]

I think it's a special case in most browsers from when I tested this last time around.

[annevk]

@achristensen07 what are your ideas here? I believe @valenting wanted to do more decoding in Gecko as well: https://bugzilla.mozilla.org/show_bug.cgi?id=1197123.

It's basically a choice between this document, Safari, and Firefox and Chrome and Edge. Arguably what Chrome and Edge do is somewhat nicer since there is more canonicalization, but it could lead to subtle issues too.

[achristensen07]

I think I'm opposed to this change. alert(new URL("http://host/p%61th")); should show the %61, not 'a'. There are a lot of URL implementations based on old RFCs, and we should try to match their behavior as closely as possible. People sometimes want '%' in the path and often in the query, and we don't want people to have to put things like "%2525" in their URL for some user agents and not others.

[annevk]

@achristensen07 note that the old RFCs encourage normalizing these in the second paragraph of https://tools.ietf.org/html/rfc3986#section-2.3.

[annevk]

To be clear, not %25, but a lot of other ranges, including %61.

[achristensen07]

Sure enough, Chrome and Edge percent decode more things in the path, but not the query. This can bee seen with alert(new URL("http://host/%45%25?%45%25")); We definitely shouldn't touch the query and all browsers seem to agree on this.
I disagree with this proposal and rfc3986 because I think a canonicalized URL should always be untouched when reparsed. With this proposal, "http://host/%%36%31" would be canonicalized to "http://host/%61" which when reparsed would become "http://host/a" which is bad. Right now Chrome percent-encodes the first '%' in "http://host/%%36%31" which is strange, and Edge throws an exception.
It is clear that something needs to change and that we need to unite behavior. Right now, it is impossible to send "GET /%2e HTTP/1.1" as the first line of an HTTP request and I think that also needs to change. I'm not sure what the solution should be, but percent-decoding more characters isn't it.

[annevk]

Paging @foolip to see if we can get Chrome's perspective on this.

[annevk]

@valenting I suggest that before changing Gecko here we try to bring this thread to a conclusion first.

[achristensen07]

Is it common on the internet to see percent-encoded dots or other characters in the path? Why was section 2.3 added to rfc3986? Safari has not percent-decoded anything in the path, and I'm not aware of any compatibility issues, but it's possible I've missed something. A solution to this problem would be to just not percent-decode anything in the path, including %2e and %2E

[gibson042]

Question: to which of the following should "http://host/%%36%31" be equivalent?

• "http://host/%2561" (representing the non-decodable percent as a percent and recovering immediately)
• "http://host/%25%25361" (representing the non-decodable percent as a percent and recovering after the character responsible for its non-decodability)
• "http://host/%25%2536%2531" (giving up on percent decoding completely upon encountering a non-decodable percent)
• nothing (it is invalid, as in the older RFCs)
• other (e.g., "http://host/a") (please no)

[achristensen07]

Safari and Firefox both leave "http://host/%%36%31" untouched when canonicalizing. They also agree on the behavior of URLs like "http://host/%2e" and I suggest we change the spec to match this. What is the rationale behind Chrome's percent-encoding only the first percent, and percent decoding some characters but not others? It leads to some confusing behavior.

[valenting]

They also agree on the behavior of URLs like "http://host/%2e" and I suggest we change the spec to match this.

We actually tried to change it in Firefox so that we decode %2e but it got backed out due to an implementation bug. :)

[achristensen07]

I still think we should change the spec to not percent-decode anything in the path. Percent-decoding %2E makes it so there is only one thing that cannot be in the path. Percent-decoding more leads to strange reparsing behavior. Simpler is better, unless there's a good reason we need this.