Improved type definitions for JSON api responses

[Choc13]

Search Terms

fetch body json generic

Suggestion

There are actually two suggestions here that would work together. I'll present them separately.

1. A generic version of the json() method on a fetch Response with the following signature: json<T>(): Promise<T>.

2. A Json<T> type that is equivalent to T only if T contains properties that can be represented as valid JSON. Implementation below. The main thing here is to enforce that any properties of type Date in T are correctly mapped from a string in order to convert Json<T> to T.

Use Cases

The use cases for both features are pretty much the same so I'll deal with them together.
When fetching data from an API we create types to represent the data that is being returned. For example, say we have the following type:

type Foo = {
    name: string;
    date: Date;
};

And we want to fetch one from an API then we'll write a function like so:

async function getFoo(): Promise<Foo> {
    const response: Response = await fetch('https://example.com/foo');
    return await response.json();
}

The problem here is that elsewhere in the application if we try use the date property on a Foo then it won't actually be of type Date, but most likely will be a string because that is how it is often represented in JSON. This leads to runtime errors, usually along the lines of some Date method being undefined for string.

Obviously we could put this down to bad programming and recognise that we should create some type to represent the data that's actually being returned from the API, for example:

type FooDto = {
    name: string;
    date: string;
};

and then update the API function to be:

async function getFoo(): Promise<Foo> {
    const response: Response = await fetch('https://example.com/foo');
    const foo: FooDto =  await response.json() as FooDto;
    return {
        ...foo,
        date: new Date(foo)
    };
}

However, it places a large burden on the developer when this needs to be done for many types and it is still error prone if the developer forgets to create the DTO type.

In our projects we have found a better way to mitigate this problem by defining a Json<T> type like so:

export type Json<T> = 
    T extends Date ? string :
    T extends Array<(infer U)> ? JsonArray<U> :
    T extends object ? { [K in keyof T]: Json<T[K]> } :
    T;

interface JsonArray<T> extends Array<Json<T>> { }

We can then define a function get<T> along these lines:

async function get<T>(url: string): Promise<Json<T>> {
    const response: Response = await fetch(url);
    return await response.json()
}

And so our implementation of getFoo becomes:

async function getFoo(): Promise<Foo> {
    const foo: Json<Foo> = await get('https://example.com/foo');
    return {
        ...foo,
        date: new Date(foo.date)
    };
}

Basically, the upshot of having the type Json<T> is that it is now not possible to forget to supply the mapping from Json<Foo> to Foo because in this case a Json<Foo> is not directly assignable to Foo due to the Json<Foo> knowing that the date property is actually a string in the Json payload. Furthermore, there is no longer a need to mandate that a DTO type be created for every object (or every object that contains a Date property). This leads to code that is therefore less prone to errors.

From our perspective it would therefore seem to make sense if something along these lines was incorporated into lib.d.ts. Our suggestion is therefore to add the Json<T> type definition and update the json() method to have the signature json<T>(): Promise<Json<T>>.

However, we obviously realise that there might be some reservations with this implementation. The main points that spring to mind are:

1. Can we assume that all dates are represented as a string in a JSON payload? This certainly seems like the most common representation, but I'm sure there will be others out in the wild.
2. Are there other complex types like Date that suffer from this problem?
3. Obviously none of this prevents developers from having type mismatches between the client and server.

To give my two cents on this, I think that the solution proposed does not prohibit someone from just calling json<any>() which would effectively give them the existing behaviour, and in fact for backwards compatibility it would probably make sense to default the generic argument to any so that existing code of that didn't specify the generic argument would still be valid. Essentially, I think that this proposal enhances the types for most cases without regressing things in the other cases.

Anyway, I hope this can at least form the start of something useful that can be included in lib.d.ts to improve the typing of data returned from an API.

Checklist

My suggestion meets these guidelines:

• This wouldn't be a breaking change in existing TypeScript/JavaScript code
• This wouldn't change the runtime behavior of existing JavaScript code
• This could be implemented without emitting different JS based on the types of the expressions
• This isn't a runtime feature (e.g. library functionality, non-ECMAScript syntax with JavaScript output, etc.)
• This feature would agree with the rest of TypeScript's Design Goals.

[AnyhowStep]

Obviously none of this prevents developers from having type mismatches between the client and server.

This is exactly why I'm against this proposal

[richthornton]

This is exactly why I'm against this proposal

That would be ideal, but it doesn't seem possible to solve with Typescript alone as they don't 'Add or rely on run-time type information in programs, or emit different code based on the results of the type system.'

This proposal seems more about trying to improve the type definitions for the response.json() function. Currently the definition is json(): Promise<any>;, with the any allowing a cast to any other type. However, we already some situations where that cast would be wrong, due to json not supporting those value types (e.g. Date). By adding a type overload of json<TExpectedResponse>(): Promise<Json<TExpectedResponse>> the consumer of .json() would know when their type is inaccurate.

As far as I know, the primitive types which json doesn't support are Date, undefined, and function. It seems like both undefined and function get removed when serialized, and so it might be worth returning the never type for these cases. I.e

export type Json<T> = 
    T extends Date ? string :
    T extends undefined ? never :
    T extends Function ? never :
    T extends Array<(infer U)> ? JsonArray<U> :
    T extends object ? { [K in keyof T]: Json<T[K]> } :
    T;

interface JsonArray<T> extends Array<Json<T>> { }

or filtering them out from the resulting type.

I think this proposal would be especially helpful as it's an incorrect type when fetching data, where we already don't have much type information. Helping to prevent people from having runtime errors by calling date functions on a string seems extremely worthwhile.

[AnyhowStep]

Counter proposal, json() : Promise<unknown>

[Choc13]

I think there are two points emerging here that are worth picking apart.

1. No one has any guarantee at runtime that the data coming off the wire is going to match what they expected when they declared their types.
2. Json is not as complete as JavaScript in the types it can represent (Date being the obvious example).

Obviously you seem very concerned about point 1, and rightly so it's a source of many mistakes.

However, this proposal is really about addressing point 2. I put the following caveat:

Obviously none of this prevents developers from having type mismatches between the client and server.

in because I wanted to make it clear that I was not trying to address point 1 with this proposal.

You are right that the data is unknown coming off the wire, but to get any benefit out of TypeScript we have to assign a type to this data at some point. Your proposal would make it mandatory for developers to perform an explicit cast to their type, but it still doesn't do anything to help them make sure they've done this correctly. What we're proposing here is a way to provide the developer with some compile time checks around the types of data they can expect to receive in a JSON payload, such that if their target type cannot be represented in JSON, as they have defined it, it would have to be mapped.

[AnyhowStep]

Use a type validation/data mapping library like,

• https://github.com/gcanti/io-ts
• https://github.com/pelotom/runtypes
• https://github.com/anyhowstep/type-mapping
• https://github.com/hapijs/joi
• etc.

Adding <T> is just adding a hidden cast. Whether you're casting to a JSON object or some other object, a cast is a cast.

With a data mapping library, it won't even really matter what the actual response is.
The data mapping library will map the string value to a Date value.

You don't have to restrict yourself to just "valid JSON types"

---

This <T extends JSONType> thing should really be more of an external library thing,

function dangerouslyPretendIsJsonType<T extends JSONType> (mixed : unknown) : T {
  return mixed as T;
}

/*snip*/
.then((response) => {
  return response.json().then(json => dangerouslyPretendIsJsonType<MyJsonType>(json));
})
.then((json) => {
  //Do stuff with strongly typed, but dangerously casted, `json`
})

[RyanCavanaugh]

As noted, there are some very nice libraries available these days for enforcing (yay!) the shape of JSON data off the wire, as well as some options for generating .d.ts from a JSON schema.

In terms of adding this definition to the library, we think this kind of definition best belongs in user space as people tend to have a lot of different opinions about what constitutes valid and what doesn't. On balance the overall gained value is IMO not that great since the restrictions around which types are JSONable and which aren't isn't that hard to internalize.

[richthornton]

@RyanCavanaugh thanks for your response, and all the great work that you do on Typescript!

It seems like there are two scenarios:

1. The ideal scenario: people use a type validation library, and json() : Promise<unknown> is the return type.

2. People do implicit casts from response.json() to their domain types.

I actually agree with @AnyhowStep's proposal to change the type to json() : Promise<unknown>. However, I imagine that the Typescript team would be reluctant to change it, as it would be a breaking change for the users doing implicit casts to their domain types.

Given these users doing an implicit cast already, it seems like it would benefit them to show where the type they are trying to cast to is impossible, given the json spec. It seems like the original mapped type was too opinionated as to date serialisation. If it was instead:

export type Json<T> = 
    T extends Date ? unknown : // instead of string
....

this would be agnostic of how they serialise their Dates, but also show them that the type they are expecting is impossible.

the restrictions around which types are JSONable and which aren't isn't that hard to internalize

I agree that the restrictions aren't hard to internalise (if you know they exist), but I also doubt that developers always check their domain models for any dates before casting.

[Choc13]

@richthornton I agree with your points and that whilst it's not hard to internalise the JSON spec, one of the main advantages of a type system is that it puts knowledge back in the world rather than requiring it to be in the developer's head. Even small amounts of individual knowledge eventually accumulate over the size of an entire program.

From the JSON spec do you not think the following definition would be more correct?

export type Json<T> =
    T extends boolean | number | string | null ? T :
    T extends Array<(infer U)> ? JsonArray<U> :
    T extends object ? { [K in keyof T]: Json<T[K]> } :
    never;

interface JsonArray<T> extends Array<Json<T>> { }

As we know that JSON can never represent anything except these primitive types, or arrays and objects of them.

I respect the original decision to close this issue as I understand the conservative viewpoint of the TypeScript team, but I just wanted to include this for posterity.