PathVariable encoding fails if request charset is not set [SPR-11474]

[spring-projects-issues]

Mats Svensson opened SPR-11474 and commented

Code below

// this works, tagId is properly decoded
@RequestMapping(value = "/tag/{tagId}", method = RequestMethod.PUT)
public @ResponseBody SourceTag updateTag(@RequestBody SourceTag newTag, @PathVariable("tagId") String tagId) {
    return sourceService.updateTag(new SourceTag(tagId), newTag);
}
// but here it fails!
@RequestMapping(value = "/tag/{tagId}", method = RequestMethod.DELETE)
public @ResponseBody SourceTag updateTag(@PathVariable("tagId") String tagId) {
    // something
}

When trying to access
/tag/mmm%C3%A6%C3%A6%C3%B8%C3%A5 as PUT with an object, the
pathvariable tagId contains the value "mmmææøå"

But when accessing
/tag/mmm%C3%A6%C3%A6%C3%B8%C3%A5 as DELETE
the pathvariable tagId contains the value "mmmææøå"

After testing, the problem seems not to be using DELETE instead of PUT, but having a @RequestBody and not having one.

---

Affects: 3.2.8, 4.0.2

Issue Links:

• Better handling of URI encoding configuration [SPR-11925] #16543 Better handling of URI encoding configuration
• Form parameters should be in UTF-8 encoding [SPR-11779] #16401 Form parameters should be in UTF-8 encoding

Referenced from: commits spring-attic/spring-framework-issues@d2cb761, spring-attic/spring-framework-issues@1e4ef2b

[spring-projects-issues]

Stéphane Nicoll commented

I have tried both use cases and they work for me. Looking at the code, there is no difference I can see if there is a @RequestBody or not.

What triggers a proper resolution of the path variable is the encoding of the request. If that is not set, Spring fallbacks on the default encoding ISO-8859-1. The result that you get in your second test is a IS0-8859-1 decoded string so I suspect that your first test sets the UTF-8 encoding while the second does not.

Also, your servlet container should be able to properly decode the URI with UTF-8. If you're using Tomcat, this is not enabled by default. e.g.

<Connector port="8080" protocol="HTTP/1.1"
    connectionTimeout="20000"
    redirectPort="8443"
    URIEncoding="UTF-8" />

Which servlet container are you using?

Please let me know what I've missed.

[spring-projects-issues]

Mats Svensson commented

I'm using Jetty for development.

You are right. Looking at the requests, sending a request with a body will add the header:
Content-Type application/json;charset=utf-8

And that's probably the difference.

However, it was my understanding that that header only should affect the body:

The purpose of the Content-Type field is to describe the data contained in the body (http://www.w3.org/Protocols/rfc1341/4_Content-Type.html)

and as such, different frameworks (for instance $http in AngularJS) I have tried actually ignore and remove Content-Type headers if there is no body in the request.

[spring-projects-issues]

Stéphane Nicoll commented

You are right regarding the Content-type but that's not what it is about.

I've created a project for you to try with your environment. Could you please get the code and validate it matches your problem?

I have tried this with Jetty8 and Jetty9 and both are working for me with the following command:

curl -X DELETE http://localhost:8080/tag/mmm%C3%A6%C3%A6%C3%B8%C3%A5

I get the following json:

{"tagId":"mmmææøå"}

The reason why this is working for me is because the URI is properly decoded by the servlet container. More specifically, HttpServletRequest#getServletPath is able to decode your URI as "mmmææøå" and not as "mmmææøå" (which would be the result with latin decoding).

If you take tomcat for instance, URI decoding is latin by default. I've fixed my local config to add URIEncoding="UTF-8" to the connector and I get the same behaviour as jetty.

-Setting a Content-type header does not change anything. The @PathVariable arguments are resolved against the URI. It seems that in your case, the servlet container fails to decode the URI.-

-Can you try this on your end and give the exact jetty version you are using?-

Thanks!

[spring-projects-issues]

Stéphane Nicoll commented

Actually, Rossen Stoyanchev and I investigated this further and we believe this has indeed a link with the charset. The updated project now reproduces the issue you're having.

To confirm this, could you please tell us how your servlet is mapped? Besides, does your Spring MVC configuration overrides RequestMappingHandlerMapping and more specifically the alwaysUseFullPath or urlDecode properties?

In the meantime, you can force the charset regardless of the request using the CharacterEncodingFilter

[spring-projects-issues]

Mats Svensson commented

Thanks for taking the time looking into this. :)

On my original project, this gives the right result, mmmææøå:
curl -X DELETE -H "Content-Type:application/json; charset=UTF8" http://localhost:8080/api/source/tag/mmm%C3%A6%C3%A6%C3%B8%C3%A5
while this one does not:
curl -X DELETE http://localhost:8080/api/source/tag/mmm%C3%A6%C3%A6%C3%B8%C3%A5

This is the servlet mapping I believe:

    <servlet>
        <servlet-name>momusapi</servlet-name>
        <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <!-- Which urls to be handled by the servlet -->
    <servlet-mapping>
        <servlet-name>momusapi</servlet-name>
        <url-pattern>/api/*</url-pattern>
    </servlet-mapping>

It's the only one, all other requests are going to static files.
(full file here: https://github.com/Studentmediene/Momus/blob/develop/src/main/webapp/WEB-INF/web.xml )

I don't think we're doing any changes to RequestMappingHandlerMapping, we're just using mvc:annotation-driven to setup everything, only adding a custom message-converter.

The jetty version is 8.1.9.v20130131 run through jetty-maven-plugin.

Thanks for the encoding filter. Currently I have a workaround just sending it as an object through put instead of delete. Not as pretty as it could have been, but ok. It's probably the only time there will be issued delete's to stuff other than numbers in the system anyway.

[spring-projects-issues]

Stéphane Nicoll commented

I haven't found anything in the spec that would indicates the contrary but ServletRequest#getCharacterEncoding only impacts the body of the request and not the URI itself so it seems that UrlPathHelper does not respect the spec.

Returns the name of the character encoding used in the body of this request. This method returns null if the request
does not specify a character encoding

What UrlPathHelper has is the notion of default encoding to use for URL decoding and this default is always overridden by the character encoding of the request if it's set. It shouldn't. Or it's probably a way to figure out the proper encoding of the whole request. In your case, not giving anything fallbacks to latin and that's what the body should use indeed (per spec).

I have pushed an implementation proposal in 456d857 but this can't go in 4.0.3 as the impact is too high. Another solution is to keep the code that we have now and add an additional option that is more strict and does not look for ServletRequest#getCharacterEncoding but as determineEncoding can be overridden, it can be hard to track what is actually going on.

Changing the default in 4.0.3 seems a bad idea. In the mean time, CharacterEncodingFilter is the way to go.

[spring-projects-issues]

Stéphane Nicoll commented

Well, this is much harder than we thought. We got our facts and those tell us that changing such behaviour would lead to a lot of nasty side effects as users are actually relying on the body encoding to decode the URI with a different encoding.

This issue helped us realized it was a larger problem and we've created #16543 to track it. Thanks for the report.