non-ASCII character in header on redirect

[aspnet-hello]

From @yellowsix on Tuesday, November 21, 2017 3:53:32 PM

I am getting an error on my website with non-ASCII characters ending up in an http header, and Kestrel does not like that. My website has URLs with non-ASCII characters, but I never set headers with such characters. Inspecting the std out logs for my site, I see exceptions like this:

fail: Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware[0]
      An unhandled exception has occurred: Invalid non-ASCII or control character in header: 0x00E1
System.InvalidOperationException: Invalid non-ASCII or control character in header: 0x00E1
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameHeaders.ThrowInvalidHeaderCharacter(Char ch)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameHeaders.ValidateHeaderCharacters(String headerCharacters)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameHeaders.ValidateHeaderCharacters(StringValues headerValues)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameResponseHeaders.SetValueFast(String key, StringValues value)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.FrameHeaders.Microsoft.AspNetCore.Http.IHeaderDictionary.set_Item(String key, StringValues value)
   at Microsoft.AspNetCore.Http.Internal.DefaultHttpResponse.Redirect(String location, Boolean permanent)
   at Microsoft.AspNetCore.Mvc.Internal.RedirectResultExecutor.Execute(ActionContext context, RedirectResult result)
   at Microsoft.AspNetCore.Mvc.RedirectResult.ExecuteResult(ActionContext context)
   at Microsoft.AspNetCore.Mvc.ActionResult.ExecuteResultAsync(ActionContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeResultAsync>d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResultFilterAsync>d__24.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResultExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeNextResourceFilter>d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeFilterPipelineAsync>d__17.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.<InvokeAsync>d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Builder.RouterMiddleware.<Invoke>d__4.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.<Invoke>d__6.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Diagnostics.StatusCodePagesMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.GetResult()
   at Microsoft.AspNetCore.Diagnostics.ExceptionHandlerMiddleware.<Invoke>d__6.MoveNext()

It seems that when a redirect happens, the code in Microsoft.AspNetCore.Http.Internal.DefaultHttpResponse.Redirect is setting a Location header, which is probably one of these URLs with a non-ASCII character...

I could be misunderstanding what is really going on here... but assuming I am not, how can I avoid this problem?

Copied from original issue: aspnet/HttpAbstractions#971

[aspnet-hello]

From @Tratcher on Tuesday, November 21, 2017 3:57:57 PM

It looks like your MVC action is returning Redirect(url), correct? How are you generating that url. Urls must be properly encoded when generated, the Redirect methods don't do it for you. Some of the encoding must be done before combining the components or else there are ambiguities.

[aspnet-hello]

From @yellowsix on Tuesday, November 21, 2017 5:15:35 PM

I see... thanks, that put me on the right track -- I'm essentially using the default ASP.NET Core 2.0 web application template with user accounts. When redirected to the Login page, it passes around a return URL as a query parameter. When that return URL is passed to the Login page, it is no longer encoded. So using that URL to redirect after a successful login will fail if it has a wacky character in it.

It seems that a call to something to escape returnUrl before using it fixes the problem -- I passed it into a Uri object and used AbsoluteUri for my purposes. Maybe something like that should be added to the RedirectToLocal method in the default visual studio template's AccountController?

[aspnet-hello]

From @Tratcher on Tuesday, November 21, 2017 6:49:25 PM

About here?
https://github.com/aspnet/templating/blob/dd8aef8699d911e48d4e7e6f4de6ba9e6860aaa2/src/Microsoft.DotNet.Web.ProjectTemplates/content/StarterWeb-CSharp/Areas/Identity/Controllers/AccountController.cs#L70

@HaoK

[JeanCollas]

What is wrong with this?

var linkLoginAndBack = $"/Account/Login?returnUrl=" +System.Net.WebUtility.UrlEncode("/Kraków");

<a href="@linkLoginAndBack">login</a>

It generates

<a href="/Account/Login?returnUrl=%2FKrak%C3%B3w" rel="nofollow">login</a>

It redirects in the browser to /Account/Login?returnUrl=%2FKraków instead of /Account/Login?returnUrl=%2FKrak%C3%B3w
How should it be done to redirect correctly?

I have tried this, but the result is the same:

Microsoft.AspNetCore.WebUtilities.QueryHelpers.AddQueryString($"/Account/Login", new Dictionary<string, string>() { { "returnUrl", "/Kraków" } })

Should it be encoded twice?

[Tratcher]

What do you see in a Fiddler trace? The browser often shows you a more human readable version than it sends on the wire.

[muratg]

@JeanCollas Any chance you can share the Fiddler trace?

[JeanCollas]

Hi,
Sorry I spent a lot of time trying to fix this, and I don't get the issue anymore.

I noticed that Redirect and RedirectLocal did not behave the same way on that issue.
I think the issue was caused by the referer header which received unencoded URL.

At the end I ended using this solution in the POST login method, that is probably not optimal at all but works for me:

                    if (urlHelper.IsLocalUrl(returnUrl))
                        return RedirectToLocal(returnUrl);
                    if (!String.IsNullOrEmpty(returnUrl))
                    {
                        try
                        {
                            if (returnUrl.StartsWith("%"))
                                returnUrl = returnUrl.UrlDecode();

                            if (returnUrl.StartsWith("/"))
                                return RedirectToLocal(returnUrl);

                            var host = (new Uri(returnUrl)).Host;

                            if (host?.EndsWith("." + SiteDomain) ?? false)
                                return Redirect(returnUrl);
                        }
                        catch { }
                    }
                    return Redirect("/");

[aspnet-hello]

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.