Object (de)serialization support with Utf8JsonReader\Writer - entry point and options

[steveharter]

This is the API proposal and feature set for object (de)serialization. It covers the main API entry point (JsonSerializer), the options (JsonSerializerOptions) and the ability to ignore properties during (de)serialization.

Review process update: due to the size of this issue and since new API issues are being added for new features, the overview and forward-looking information has been moved to https://github.com/dotnet/corefx/blob/master/src/System.Text.Json/docs/SerializerProgrammingModel.md. Future API additions will have their own independent API issue created instead of re-using this issue.

Current Status

The reviewed portion of the API is in corefx master and Preview 4. It consists of the JsonSerializer and the JsonSerializerOptions class and provides no extensibility features.

There is a previous prototype at CoreFxLab in the package System.Text.JsonLab.Serialization. It contains non-reviewed APIs including extensibility features like using attributes to define custom value converters, property name policies (like camel-casing), etc. It is build against .NET Core 3.0 Preview 2.

The last status of the API review is shown below.

API

JsonSerializer

This static class is the main entry point.

Let's start with coding examples before the formal API is provided.

Using a simple POCO class:

    public class Person
    {
        public string FirstName { get; set; }
        public string LastName { get; set; }
        public DateTime? BirthDay { get; set; }
    }

To deserialize JSON bytes into a POCO instance:

    ReadOnlySpan<byte> utf8= ...
    Person person = JsonSerializer.Parse<Person>(utf8);

To serialize an object to JSON bytes:

    Person person = ...
    byte[] utf8 = JsonSerializer.ToBytes(person);

The string-based Parse() and ToString() are convenience methods for strings, but slower than using the <byte> flavors because UTF8 must be converted to\from UTF16.

namespace System.Text.Json.Serialization
{
    public static class JsonSerializer
    {
        public static object Parse(ReadOnlySpan<byte> utf8Json, Type returnType, JsonSerializerOptions options = null);
        public static TValue Parse<TValue>(ReadOnlySpan<byte> utf8Json, JsonSerializerOptions options = null);

        public static object Parse(string json, Type returnType, JsonSerializerOptions options = null);
        public static TValue Parse<TValue>(string json, JsonSerializerOptions options = null);

        public static ValueTask<object> ReadAsync(Stream utf8Json, Type returnType, JsonSerializerOptions options = null, CancellationToken cancellationToken = default(CancellationToken));
        public static ValueTask<TValue> ReadAsync<TValue>(Stream utf8Json, JsonSerializerOptions options = null, CancellationToken cancellationToken = default(CancellationToken));

        public static byte[] ToBytes(object value, Type type, JsonSerializerOptions options = null);
        public static byte[] ToBytes<TValue>(TValue value, JsonSerializerOptions options = null);

        public static string ToString(object value, Type type, JsonSerializerOptions options = null);
        public static string ToString<TValue>(TValue value, JsonSerializerOptions options = null);

        public static Task WriteAsync(object value, Type type, Stream utf8Json, JsonSerializerOptions options = null, CancellationToken cancellationToken = default(CancellationToken));
        public static Task WriteAsync<TValue>(TValue value, Stream utf8Json, JsonSerializerOptions options = null, CancellationToken cancellationToken = default(CancellationToken));
    }
}

JsonSerializerOptions

This class contains the options that are used during (de)serialization.

If an instance of JsonSerializerOptions is not specified when calling read\write then a default instance is used which is immutable and private. Having a global\static instance is not a viable feature because of unintended side effects when more than one area of code changes the same settings. Having a instance specified per thread\context mechanism is possible, but will only be added pending feedback. It is expected that ASP.NET and other consumers that have non-default settings maintain their own global, thread or stack variable and pass that in on every call. ASP.NET and others may also want to read a .config file at startup in order to initialize the options instance.

An instance of this class and exposed objects will be immutable once (de)serialization has occurred. This allows the instance to be shared globally with the same settings without the worry of side effects. The immutability is also desired with a future code-generation feature. Due to the immutable nature and fine-grain control over options through Attributes, it is expected that the instance is shared across users and applications. We may provide a Clone() method pending feedback.

For performance, when a JsonSerializerOptions instance is used, it should be cached or re-used especially when run-time attributes are added because when that occurs, caches are held by the instance instead of being global.

namespace System.Text.Json.Serialization
{
    /// <summary>
    /// Provides options to be used with <see cref="JsonSerializer"/>.
    /// </summary>
    public class JsonSerializerOptions
    {
        public JsonSerializerOptions();

        /// <summary>
        /// Defines whether an extra comma at the end of a list of JSON values in an object or array
        /// are allowed (and ignored) within the JSON payload being read.
        /// By default, it's set to false, and the reader will throw a <exception cref="JsonReaderException"/> if it encounters a trailing comma.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public bool AllowTrailingCommas {get; set;}

        /// <summary>
        /// The default buffer size in bytes used when creating temporary buffers.
        /// </summary>
        /// <remarks>The default size is 16K.</remarks>
        /// <exception cref="System.ArgumentException">Thrown when the buffer size is less than 1.</exception>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public int DefaultBufferSize { get; set; }

        /// <summary>
        /// Determines whether null values are ignored during serialization and deserialization.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public bool IgnoreNullValues { get; set; }

        /// <summary>
        /// Determines whether readonly properties are ignored during serialization and deserialization.
        /// A property is readonly if it contains a public getter but not a public setter.
        /// The default value is false.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public bool IgnoreReadOnlyProperties { get; set; }

        /// <summary>
        /// Gets or sets the maximum depth allowed when reading or writing JSON, with the default (i.e. 0) indicating a max depth of 64.
        /// Reading past this depth will throw a <exception cref="JsonReaderException"/>.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public int MaxDepth { get; set; }

        /// <summary>
        /// Defines how the <see cref="Utf8JsonReader"/> should handle comments when reading through the JSON.
        /// By default the reader will throw a <exception cref="JsonReaderException"/> if it encounters a comment.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public JsonCommentHandling ReadCommentHandling { get; set; }

        /// <summary>
        /// Defines whether the <see cref="Utf8JsonWriter"/> should pretty print the JSON which includes:
        /// indenting nested JSON tokens, adding new lines, and adding white space between property names and values.
        /// By default, the JSON is written without any extra white space.
        /// </summary>
        /// <exception cref="InvalidOperationException">
        /// Thrown if this property is set after serialization or deserialization has occurred.
        /// </exception>
        public bool WriteIndented { get; set; }

        // several more options here for other features not covered here
    }

    /// <summary>
    /// The base class of serialization attributes.
    /// </summary>
    public abstract class JsonAttribute : Attribute
    {
    }

    /// <summary>
    /// Prevents a property from being serialized or deserialized.
    /// </summary>
    [AttributeUsage(AttributeTargets.Property, AllowMultiple = false)]
    public sealed class JsonIgnoreAttribute : JsonAttribute
    {
        public JsonIgnoreAttribute();
    }
}

[Tornhoof]

Thank you for describing your API and performance concepts.
API

• I'd like to see non-generic FromJson methods, e.g. for the ASP.NET Core TextFormatter classes, assuming this will not change that much, see https://github.com/aspnet/AspNetCore/blob/321327f84b2b22dcff2e9beb06a9a64236c5cced/src/Mvc/src/Microsoft.AspNetCore.Mvc.Core/Formatters/TextInputFormatter.cs
• What about global settings in JsonConverterSettings, i.e. NoNulls, Int/String Enums etc?

Performance

No objects created on the heap during (de)serialization after warm-up, with the exception of the POCO class itself and related properties during serialization.

This probably won't be true if you want to support things like ReadOnlyCollection, I probably would simply say "related properties and types".

Optimized hashtable lookup for property names using byte[] as the key.

This will be slow, I recommend talking to @rynowak about his work in ASP.NET Core Routing and the IL Matcher (the code is pretty much what you need, technically matching a route and matching a property name is quite similar), this is by far the fastest approach, in both the Routing and all the experiments I did for SpanJson https://github.com/Tornhoof/SpanJson/wiki/Technology-and-Internals#3-comparison-against-integers (which is pretty much identical to the approach in Routing).
I expect that you could probably reuse 75%+ of the Routing code (an estimation based on how long it took to port my Expression Tree Code from SpanJson to do the Routing API, which was trivial).

[davidfowl]

Where’s the async API?

[ahsonkhan]

Where’s the async API?

In what scenario would you require async (de)serialize APIs? Can you please provide a sample usage?

[davidfowl]

I'm going to be a bit obtuse here. If we're not making these APIs async then we need to have a long discussion again about ASP.NET Core usage of these APIs.

Input:

https://github.com/aspnet/AspNetCore/blob/master/src/Mvc/src/Microsoft.AspNetCore.Mvc.Formatters.Json/JsonInputFormatter.cs#L318

Output:

https://github.com/aspnet/AspNetCore/blob/ca7c48c520f2baae032e0429f19552abed77e009/src/Mvc/src/Microsoft.AspNetCore.Mvc.Formatters.Json/JsonOutputFormatter.cs#L147-L152

Just to throw out some more examples here's the HttpClient JSON formatter that's used today in most applications:

https://github.com/aspnet/AspNetWebStack/blob/39d3064baf13181f5718ec5d85ba644b47d0704b/src/System.Net.Http.Formatting/Formatting/BaseJsonMediaTypeFormatter.cs#L189

To mitigate sync reading, today we buffer the entire Stream into memory or disk (depending on the size):

https://github.com/aspnet/AspNetCore/blob/ca7c48c520f2baae032e0429f19552abed77e009/src/Mvc/src/Microsoft.AspNetCore.Mvc.Formatters.Json/JsonInputFormatter.cs#L240-L249

Which is unfortunate and we'd really love to avoid that. The plan was to have to get that benefit from the new JSON reader/writer/serializer. We don't have a good solution for output today. You end up doing synchronous IO once you hit a buffer threshold which ends up doing sync over async and leans towards causing thread pool starvation.

See dotnet/aspnetcore#6397 for more details on output. ASP.NET Core is going to disable all sync IO on the request and response by default in 3.0.

[Tornhoof]

For the ASP.NET Core formatter you additionally need to support a bag of json properties, which is either serialized into the parent object or deserialized into the bag for RFC 7807 Error Details. See aspnet/Mvc#8529 for more details, that one uses JSON.NET's https://www.newtonsoft.com/json/help/html/T_Newtonsoft_Json_JsonExtensionDataAttribute.htm

[ahsonkhan]

cc @KrzysztofCwalina, @jkotas, @glennc, @terrajobst as an FYI (note: this is still WIP)

[rynowak]

My feedback in a rough stream-of-consciousness format.

On JsonConverter, we will need the ability to specify the type dynamically as an argument - in addition to the generic overloads. Without this, it will really cause a lot of folks to use a workaround to call the generic version with MakeGenericMethod. It's OK to box in this case.

JsonConverterSettings seems like it has a mashup of settings that generally global to an operation, and don't pertain to specific object or type (indent size for example). Is it expected that I can specify a global indent size, but then override the indent size for a specific type?

I'm really intrigued by how JsonConverterSettings tries to maintain a loose coupling with the specific types of settings/options we provide support for. This is similar to a number of things we do in MVC and it scales horizontally pretty nicely.

I think it would be really neat to see a recipe guide or some side-by-side examples: "this is how you'd set the json property name with an attribute vs with code".

Another natural thing that users will want from JsonConverterSettings is a copy constructor and a way to merge them.

The JsonClassMaterializer seems like a god-object. This is where you control all object creation and property get/set operations for all objects and all types. If the intention is to use these as a replacement for JSON.Net's JsonConverter then I think it's far off. This seems like something that would be intentionally hard to replace. Maybe that's OK depending on what other things we do.

---

Async is a higher-level bugaboo (I think that's a SFW replacement for what I was thinking 😆). I still think C# and .NET do async better than any other language out there, but we're still struggling to pull together an end-to-end here for the web that actually avoids synchronous I/O at all levels.

From my perspective, I'm still excited to see all of the work that's here. I think we're solving a very critical problem and these are all steps in the right direction.

[steveharter]

@Tornhoof

I'd like to see non-generic FromJson methods, e.g. for the ASP.NET Core TextFormatter classes, assuming this will not change that much

Yes that makes sense. The API is still being defined, and this issue wasn't meant to be complete at this point.

What about global settings in JsonConverterSettings, i.e. NoNulls, Int/String Enums etc?

Currently those simple settings are in JsonPropertyAttribute which can be "added" at run-time. If too verbose we may just place those right on JsonConverterSettings.

Performance

No objects created on the heap during (de)serialization after warm-up, with the exception of the POCO class itself and related properties during serialization.

This probably won't be true if you want to support things like ReadOnlyCollection, I probably would simply say "related properties and types".

Yes, of course.

Optimized hashtable lookup for property names using byte[] as the key.

This will be slow, I recommend talking to @rynowak about his work in ASP.NET Core Routing and the IL Matcher (the code is pretty much what you need, technically matching a route and matching a property name is quite similar), this is by far the fastest approach, in both the Routing and all the experiments I did for SpanJson https://github.com/Tornhoof/SpanJson/wiki/Technology-and-Internals#3-comparison-against-integers (which is pretty much identical to the approach in Routing).

Thanks. I updated the description and implementation to use an interger-based array which is looking very promising at this point in time.

[steveharter]

@rynowak

On JsonConverter, we will need the ability to specify the type dynamically as an argument - in addition to the generic overloads. Without this, it will really cause a lot of folks to use a workaround to call the generic version with MakeGenericMethod. It's OK to box in this case.

Yes that will be added. Thanks

JsonConverterSettings seems like it has a mashup of settings that generally global to an operation, and don't pertain to specific object or type (indent size for example). Is it expected that I can specify a global indent size, but then override the indent size for a specific type

As it is now, JsonConverterSettings is fairly clean and only contains settings that would need to be specified at run-time like MaxDepth and DefaultBufferSize (I'll update the API doc soon with new members). The mashup is JsonPropertyAttribute for now.

I'm really intrigued by how JsonConverterSettings tries to maintain a loose coupling with the specific types of settings/options we provide support for. This is similar to a number of things we do in MVC and it scales horizontally pretty nicely.

Yes there is some loose coupling with the extension points like the property casing where you can provide the implementation. Other simple settings will not really be extensible in that way but the values can be specified via attributes at the assembly, class and property level plus run-time overrides to each of these -- i.e. the values for these simple settings can be changed in many flexible ways, but not necessarily the code that uses them.

I think it would be really neat to see a recipe guide or some side-by-side examples: "this is how you'd set the json property name with an attribute vs with code".

Yes I will do this in the next update.

The JsonClassMaterializer seems like a god-object. This is where you control all object creation and property get/set operations for all objects and all types. If the intention is to use these as a replacement for JSON.Net's JsonConverter then I think it's far off. This seems like something that would be intentionally hard to replace. Maybe that's OK depending on what other things we do.

JsonClassMaterializer is just an enum. The implementation is internal. We may not even need the enum if we decide the default is ok. Currently the default will attempt to use the most efficient mechanism (currently IL gen for get,set,ctor) but if IL gen is not supported then use standard reflection. However, the enum is there currently for someone to change the default if for example they have a single object to deserialize and don't want any additional memory overhead (and initial perf hit) of IL so they want to use standard reflection instead.

[steveharter]

@davidfowl

I'm going to be a bit obtuse here. If we're not making these APIs async then we need to have a long discussion again about ASP.NET Core usage of these APIs.

I added the async apis along with a brief description. Currently they only support a Stream, more if necessary. I have a prototype that verifies the feasibility.

[KrzysztofCwalina]

Some quick feedback:

1. As @davidfowl have said, we need async APIs that can work with Stream and Pipelines
2. I don't think the serializer/converter should have a public dependency on JsonReader. The reader should be an implementation detail. The reasons are: a) the reader is a very tricky type that most our users should not be concerned with, b) we should preserve the ability change the implementation to other than the reader.
3. The deserailizer/converter should be in a separate namespace from the low level reader/writer types. I don't think we should expose the average .NET developer to the complexity of the readers (e.g. to by-ref types).
4. We should decide if cancellation tokes are optional or not.
5. I would love to have the ability to generate deserialization code at build time and include it in my projects. In Azure SDK we almost always know the shape of the deserialized schema and so emiting the deserialization code at design time would:
   • make it more efficient
   • be more AOT friendly
   • allow the deserialized types to appear immutable (as the mutators could be internal visibility).

[Tornhoof]

@steveharter If you care about integer tricks, you can optimize the json member writing for utf8 with the same trick. Assuming you will bake precalculated byte-arrays for the property names into the IL, you can use the same concept as for deserialization. Instead of copying the precalculated buffer to the output, just write appropriate integer values to the output.
Instead of writing Encoding.UTF8.GetBytes("\"message\":"), you could also write

jsonWriter.WriteUtf8Verbatim(7306916068917079330UL /*"message*/, 14882 /*":*/);

Atleast in my SpanJson tests that was the last main difference in serialization speed between UTF16 and UTF8 as, atleast for expression trees, there is a large difference if I have
Expression.Constant("\"message:\""); which is fast for utf16 or Expression.Constant(Encoding.UTF8.GetBytes("\"message\":")) for UTF8 which was ~20% slower than the UTF16 version.
In SpanJson properties up to ~30 chars length are written via integers, for longer properties the byte[] copy (incl. expression tree) overhead is pretty much the same speed or faster
This might not be necessary if you can leverage those shiny new static byte arrays with public ReadOnlySpan<byte> property => new byte[]{0xA,0xB,0xC}, but it might be worth a try.

That's more or less the last larger performance trick in SpanJson as now the UTF8 and UTF16 serialization speeds are the same, still deserialization of UTF8 is slower, but that's mostly due to the utf8 to utf16 overhead for strings., especially since @stephentoub did all those nice TryFormat``TryParse`` optimizations for both UTF8 and UTF16.

[KrzysztofCwalina]

An additional thing we would need to support Azure SDK: version resilient serialization, including round-tripping.

We need to be able to add a property bag field (dictionary) to deserialized types. If deserialized payload has a property that does not exist in the type, the property would be saved in the dictionary. The serializer would serialize not only actual properties of the type, but also the dictionary (so all payload round trips).

cc: @schaabs

[pranavkm]

@KrzysztofCwalina @Tornhoof's comment from here - https://github.com/dotnet/corefx/issues/34372#issuecomment-451928379 - goes in to this. JSON.NET supports it using ExtensionDataAttribute, the data contract serializers support it using IExtensibleDataObject.