HttpContext and JSON

[rynowak]

HttpContext and JSON

This is a proposal to add extension methods for using HttpContext (and related types) with System.Text.Json. Adding this functionality will make ASP.NET Core developers productive with smaller amounts of code. This is aligned with our goal of making route-to-code a useful programming pattern.

endpoints.MapPost("/weather", async context =>
{
    var weather = await context.Request.ReadFromJsonAsync<WeatherForecast>();
    await UpdateDatabase(weather);

    context.Response.StatusCode = StatusCodes.Status202Accepted;
});

Goals

• Make this the easiest imperative way to read/write JSON

• Make this fastest way to read/write JSON

• Do HTTP things correct by default (don't cut corners)

• Replace the guts of MVC's JSON formatter with this (validates flexibility)

• Do all of this without requiring service registration/configuration

• NON-GOAL: Create an abstraction for JSON serializers/settings

API Additions

These are the proposed APIs, see the following (annotated) sections for discussion.

Some updates based on recent discussions with @JamesNK and @Tratcher:

• These would be added to the Microsoft.AspNetCore.Http.Extensions assembly.
• JsonOptions would be inherited (used as the base) by MVC and SignalR - but they are copies. So modifying JsonOptions would influence MVC and SignalR - but not the reverse. This is worth a discussion.
• We got rid of APIs that allow specifying an Encoding for writing. Can bring these back if needed.
• Overloads that specify a contentType should specify the value they want to see in the Content-Type header. We will not append a charset. We will not parse the value you give us and change the encoding based on charset. It's your job to get it right.

namespace Microsoft.AspNetCore.Http
{
    public static class HttpRequestJsonExtensions
    {
        public static bool HasJsonContentType(this HttpRequest request);
    }
}

namespace Microsoft.AspNetCore.Http.Json
{
    public static class HttpContextJsonExtensions
    {
        public static ValueTask<TValue> ReadFromJsonAsync<TValue>(
            this HttpRequest request, 
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask<TValue> ReadFromJsonAsync<TValue>(
            this HttpRequest request, 
            JsonSerializerOptions? options, 
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask<object> ReadFromJsonAsync(
            this HttpRequest request, 
            Type type, 
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask<object> ReadFromJsonAsync(
            this HttpRequest request, 
            Type type, 
            JsonSerializerOptions? options, 
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask WriteAsJsonAsync<TValue>(
            this HttpResponse response, 
            TValue value, 
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask WriteAsJsonAsync<TValue>(
            this HttpResponse response, 
            TValue value, 
            JsonSerializerOptions? options,
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask WriteAsJsonAsync<TValue>(
            this HttpResponse response,
            TValue value,
            JsonSerializerOptions? options,
            string? contentType,
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask WriteAsJsonAsync(
            this HttpResponse response,
            Type type,
            object? value,
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask WriteAsJsonAsync(
            this HttpResponse response,
            Type type,
            object? value,
            JsonSerializerOptions? options,
            CancellationToken cancellationToken = default) { throw null; }

        public static ValueTask WriteAsJsonAsync(
            this HttpResponse response,
            Type type,
            object? value,
            JsonSerializerOptions? options,
            string? contentType,
            CancellationToken cancellationToken = default) { throw null; }
    }

    public class JsonOptions
    {
        public JsonSerializerOptions SerializerOptions { get; }
    }
}

Design Notes

0: General Shape

There are three primary APIs here:

• Detect if the request body is JSON
• Read JSON from the body
  • Using a static type (TValue)
  • Using a dynamic type (Type type)
• Write JSON to the body
  • Using a static type (TValue value)
  • Using a dynamic type (Type type, object value)

In general I've tried to keep naming consistent with the functionality for reading the form (HasFormContentType, ReadFormAsync()).

1: HasJsonContentType

I tried to make the design of this similar to HasFormContentType which already exists on the HttpRequest. I considered as well making this an extension method, but there's no good reason to deviate from the pattern we established with form.

There's nothing about this property that makes it specific to a serializer, it's a straightforward comparison of the content type.

2: Namespace

One of the challenges in this area is that if we park the best method names to mean System.Text.Json, then this will be confusing to someone using another serializer. It would be really annoying to use JIL and see ReadJsonAsync() show up everywhere, but mean System.Text.Json. For this reason I put these extensions in a different namespace.

You could imagine that another serializer would want to provide similar functionality. I want to make sure the existing serializer ecosystem can coexist with this addition.

---

In my mind there are a few ways to address this challenge:

• Create a serializer abstraction
• Put System.Text.Json in the method names
• Make this an extra package
• Use a namespace to isolate this functionality

---

Creating a serializer abstraction has a bunch of problems:

• The abstraction would have to go in the BCL where they (likely) don't want it
• We'd likely need to require service registration (or coupling in hosting) to use these extensions
• We'd have to remove JsonSerializerOptions from the APIs

I reject this option, I think it defeats enough of our stated goals.

---

Considering other options - the namespace seems like the best, most flexible choice. Putting System.Text.Json in the names would be flexible as well, but would be ugly. So I'm pitching the namespace.

I think any of the non-abstraction options would be reasonable for us to choose while still meeting our goals.

3: ReadJsonAsync and Overload Set

I took the approach of defining lots of overloads here with just the cancellation token as an optional parameter.

The reason why is because of what happens when you using cancellation tokens with other optional parameters. Imagine that I made a single overload with optional parameters for both options and cancellation token.

public static ValueTask<TValue> ReadJsonAsync<TValue>(
    this HttpRequest request, 
    JsonSerializerOptions options = default, 
    CancellationToken cancellationToken = default) { throw null; }

await httpContext.Request.ReadJsonAsync<WeatherForecast>(myCancellationToken); // compile error

await httpContext.Request.ReadJsonAsync<WeatherForecast>(null, myCancellationToken); // works
await httpContext.Request.ReadJsonAsync<WeatherForecast>(cancellationToken: myCancellationToken); // works

We can avoid this ugliness by defining more overloads.

4: Media Type and Encoding

There are a lot of overloads of WriteJsonAsync - I'm interested in ways to cut this down.

We need to expose the media type as an option because it's becoming more common to use suffix content types like application/cloud-events+json. This fills a gap in what traditional media types do - a media type like application/json describes a data-representation, but says nothing about the kind of data being represented. More modern designs like CloudEvents will use suffix content types to describe both.

We need to expose the encoding because we MVC supports it (it's a goal to have MVC call this API), and because we still have the requirement to support things like GB2312.

I don't think either of these things are contraversial, we should debate whether we're happy with the design being proposed here as the best way to express this flexibility.

5: Managing JsonSerializerOptions

We need a strategy to deal with the fact that the default settings of JsonSerializerOptions are bad for the web. We want the serializer to output camelCase JSON by default, and be more tolerant of casing differences on input. Also because we know that we're always outputting text with a character encoding, we can be more relaxed about the set of characters we escape compared to the defaults.

I reject the idea that we'd give the user the default JsonSerializerOptions through these APIs and make it their job to manage, because that conflicts with the goal of this being the easiest way to do JSON - we want these APIs to have good defaults for the web.

There's a couple of options for how we could implement this:

• Have a static instance
• Have a feature on the http context
• Use options

Each of these have a downside. The static is a static, the downside is that its static. Using a feature either allocates a bunch or has wierd coupling (kestrel coupled to a serializer). Using options has some runtime overhead for the service lookup. Of these options seems like the best choice. We could also use the options approach to share the options instance between MVC's JsonOptions and this one for compatibility.

Behavious

HasJsonContentType

This method will return true when a request has a JSON content type, that is:

• application/json
• text/json
• application/*+json

Null or empty content type is not considered a match.

ReadJsonAsync

The overloads of ReadJsonAsync will throw an exception if the request does not have a JSON content type (similar to ReadFormAsync).

Depending on the value of CharSet the method may need to create a stream to wrap the request body and transcode - System.Text.Json only speaks UTF8. We're in discussions with CoreFx about moving our transcoding stream implementations into the BCL. We will assume UTF8 if no CharSet was provided, the serializer/reader will validate the correctness of the bytes.

We'll call the appropriate overload of JsonSerializer.DeserializeAsync to do the heavy lifting.

---

There's a couple of usability concerns here related to error handling. These APIs optimize for the happy path:

• Throws exceptions for non-JSON content type
• Throws exceptions for invalid JSON content

Someone who wants to handle both of these errors and turn them into a 400 would need to write an if for the content type, and a try/catch for the possible exceptions from deserialization.

It would be possible to make a TryXyz set of APIs as well - they would end up handling the exception for you. Since these are extension methods, they can't really log (without service locator).

With a union:

public static ValueTask<(bool success, TValue value)> TryReadJsonAsync<TValue>(this HttpRequest request);

endpoints.MapPost("/weather", async context =>
{
    var result = await context.Request.TryReadJsonAsync<WeatherForecast>();
    if (!result.success)
    {
        context.Response.StatusCode = StatusCodes.Status400BadRequest;
        return;
    }

    await UpdateDatabase(result.value);

    context.Response.StatusCode = StatusCodes.Status202Accepted;
});

A possibly better version:

public static ValueTask<TValue> ReadJsonOrDefaultAsync<TValue>(this HttpRequest request);

endpoints.MapPost("/weather", async context =>
{
    var weather = await context.Request.ReadJsonOrDefaultAsync<WeatherForecast>();
    if (weather is null)
    {
        context.Response.StatusCode = StatusCodes.Status400BadRequest;
        return;
    }

    await UpdateDatabase(weather);

    context.Response.StatusCode = StatusCodes.Status202Accepted;
});

WriteJsonAsync

Prior to doing the serialization to the response body, the WriteJsonAsync method will write a content type (with CharSet) to the Content-Type header. If no content type is specified then application/json will be used. If no encoding is specified, then UTF8 will be used.

Serialization will call JsonSerializer.SerializeAsync - and provide a wrapper stream if an encoding other than UTF8 is in use.

Code Samples

Reading JSON from the request.

endpoints.MapPost("/weather", async context =>
{
    var weather = await context.Request.ReadJsonAsync<WeatherForecast>();
    await UpdateDatabase(weather);

    context.Response.StatusCode = StatusCodes.Status202Accepted;
});

---

Writing JSON to the response.

endpoints.MapGet("/weather/{city}", async context =>
{
    var city = (string)context.Request.RouteValues["city"];
    var weather = GetFromDatabase(city);

    await context.Response.WriteJsonAsync(weather);
});

---

Writing JSON to the response with explicit content type.

endpoints.MapGet("/weather/{city}", async context =>
{
    var city = (string)context.Request.RouteValues["city"];
    var weather = GetFromDatabase(city);

    await context.Response.WriteJsonAsync(weather, options: null, mediaType: "application/weather+json");
});

I'm not completely in love with this one. We might want to think about making more parameters optional.

---

Explicitly handling bad content-type

endpoints.MapPost("/weather", async context =>
{
    if (!context.Request.HasJsonContentType)
    {
        context.Response.StatusCode = StatusCodes.Status415UnsupportedMediaType;
        return;
    }

    var weather = await context.Request.ReadJsonAsync<WeatherForecast>();
    await UpdateDatabase(weather);

    context.Response.StatusCode = StatusCodes.Status202Accepted;
});

---

Letting routing handle bad content-type (possible feature)

endpoints.MapPost("/weather", async context =>
{
    var weather = await context.Request.ReadJsonAsync<WeatherForecast>();
    await UpdateDatabase(weather);

    context.Response.StatusCode = StatusCodes.Status202Accepted;
})
.WithRequiredContentType("application/json");

[Tratcher]

1: HasJsonContentType

I tried to make the design of this similar to HasFormContentType which already exists on the HttpRequest. I considered as well making this an extension method, but there's no good reason to deviate from the pattern we established with form.

There's nothing about this property that makes it specific to a serializer, it's a straightforward comparison of the content type.

Not a fan of adding any more stuff directly on HttpRequest. HasFormContentType mainly makes sense because Form is also there, and in hindsight neither should have been.

When all of the other new JSON APIs are extensions, HasJsonContentType might as well be too.

[rynowak]

1: HasJsonContentType

I tried to make the design of this similar to HasFormContentType which already exists on the HttpRequest. I considered as well making this an extension method, but there's no good reason to deviate from the pattern we established with form.
There's nothing about this property that makes it specific to a serializer, it's a straightforward comparison of the content type.

Not a fan of adding any more stuff directly on HttpRequest. HasFormContentType mainly makes sense because Form is also there, and in hindsight neither should have been.

When all of the other new JSON APIs are extensions, HasJsonContentType might as well be too.

I think either one of these could be reasonable, I went back and forth on which one to propose. It accomplishes the same goal either way, just slightly inconsistent in terms of syntax.

[davidfowl]

While I agree it would be ideal to not add more virtuals like this to the HttpContext, extension methods for things like this aren't great. Would we still do this if we add the Try methods?

[xsoheilalizadeh]

Why the HttpResponse isn't readable and I can't see any ReadJsonAsync extension method for that in your API proposal.

[davidfowl]

Why the HttpResponse isn't readable and I can't see any ReadJsonAsync extension method for that in your API proposal.

The HttpResponse is write only. You can read the body if you replace the output stream with a readable stream (like a memory stream or something more efficient...)

[xsoheilalizadeh]

You can read the body if you replace the output stream with a readable stream (like a memory stream or something more efficient...)

I did it in this way and not sure it's efficient. Could you show me similar internal usage of reading HttpResponse in ASP.NET Core?

[jstclair]

In the second-to-last example, isn’t that mixing up Content-Type and Accept?

[Yves57]

Request content type check

if (!context.Request.HasJsonContentType)
{
    context.Response.StatusCode = StatusCodes.Status415UnsupportedMediaType;
    return;
}

seems a little bit painful to write to each endpoint.
But .WithRequiredContentType("application/json") breaks the imperative code idea for me (and there is no possibility anymore to set a custom status code).

Extension methods target

But I was asking myself if imperative involved low level. For example attach the extension methods to HttpRequest and HttpResponse is the most logical in a technical point of view, but if the extension methods were attached to HttpContext, the code would be maybe clearer to read? In the same interrogation imperative / low level, are there other methods like WriteStatusCode in the pipe (not mandatory for a low level functionality, but useful in a imperative code)?

endpoints.MapPost("/weather", async context =>
{
    if (await context.TryReadJsonAsync<WeatherForecast>(out var weather))
    {
       await UpdateDatabase(weather);

       await context.WriteJsonAsync(weather);
   }
   else
   {
      context.WriteStatusCode(StatusCodes.Status415UnsupportedMediaType);
      // await context.WriteStatusCodeAsync(StatusCodes.Status415UnsupportedMediaType);
   }
});