Typed dicts with missing keys

[JukkaL]

We don't have a complete story for typed dicts when some keys may be missing. Support for get (#2612) is necessary, but it may not be sufficient. For example, consider code like this:

A = TypedDict('A', {'x': int, 'y': str})
a: A = {'x': 1}

Should it be possible to make this type check without a cast?

[JukkaL]

There are many reasonable use cases where some typed dict keys are only defined some of the time:

• If we have a JSON config file, it's reasonable to have some optional keys there, and it's also reasonable that later versions of a program add additional keys that may be missing if an earlier program version created the config file.
• Similarly, if JSON is used in a network protocol, some keys may be optional to save some bandwidth. Also, older versions of a client may be missing some keys introduced in later clients.
• If JSON is used for serialization, different versions of the serialized object may have different attributes, and we may want to omit some keys with default values to save space.
• If we use TypedDict as an argument type, later API versions may support additional keys that are optional to retain backward compatibility.

A simple approach would be have a way to annotate potentially missing keys. By default, a key would still be required. We can't use Optional[...] for this since it has another meaning. Some ideas below:

A = TypedDict('A', {'x': OptionalKey[int]})
A = TypedDict('A', {'x': MayExist[int]})
A = TypedDict('A', {'x': NotRequired[int]})

class A(TypedDict):
    a: OptionalKey[int]
    b: MayExist[int]

This doesn't work:

A = TypedDict('A', {'x?': int})  # doesn't work with the alternative syntax

If some key is optional, it must be accessed using get() or guarded with an in check:

if 'x' in d:
    d['x']   # okay if 'x' is optional
d.get('x')  # okay if 'x' is optional
d['x']  # failure if 'x' is optional

Maybe we also need to track setting optional keys in the conditional type binder:

if 'x' not in d:
    d['x'] = 1
d['x']  # okay even if 'x' is optional, since we set it above

[JukkaL]

When receiving JSON objects from an untrusted source, we'd want to check that it conforms to our schema before using it as a typed dict. Otherwise bad things could happen. If optional key constraints are available for introspection, we could implement a helper that checks whether a runtime dictionary object is compatible with a typed dict type. This wouldn't be part of mypy or typing, but it could be a useful separate module.

[rowillia]

FWIW hacklang conflates keys with Nullable with Optional Keys - https://docs.hhvm.com/hack/shapes/introduction#accessing-fields__nullable-fields-are-optional

I wouldn't worry too much about schema validation in Mypy. Marshmallow is already great at that (And the hypothetical plugin system would allow them to play nice together 😄)

[JukkaL]

@rowillia An approach similar to Hack sounds pretty reasonable for mypy as well. Here's how it could work:

• d['x'] works no matter what type 'x' has.
• d.get('x') is only allowed if 'x' has an optional type.
• Keys with an optional type can be left out when constructing a typed dict instance.

This would clearly be unsafe, though. d['x'] could fail at runtime, in case 'x' is optional. Also, if some external library or service actually expects an explicit None value, it's easy to accidentally omit a required key and cause a runtime error.

Maybe mypy should be able to help catch these issues. Some ideas:

1. Have a flag for each typed dict specifying how optional types work. There could be three options (or maybe just two of these): 1) Optional keys can be left out, but do not enforce get (perhaps the default). 2) Optional keys can be left out, but enforce using get with optional keys. 3) Keys with optional types cannot be left out.
2. Have a config file and command line option for enabling strict get checks for all typed dicts. When this option is enabled, get must be used with optional keys. This is a pretty coarse-grained tool, but it might still work reasonably well.
3. Let users mark fields with optional types as required. The syntax could be {'x': Required[Optional[int]]}. Such a key must always be present, and using get is not allowed. By default we could fall back to the Hack-like approach. Alternative spelling: ValueRequired[Optional[...]].
4. Like (3), but enforce using get with optional keys (no d['x']).

I'm leaning towards (4), since it seems to provide sufficient flexibility with reasonable defaults, and it would also be safe. Also, I'd probably allow get to be used with arbitrary keys to make checking legacy code easier -- after all, the get call won't fail at runtime even if a key is required.

@davidfstr @gvanrossum What do you think?

[davidfstr]

A few quick thoughts:

• Agreed that enabling external format checkers to check JSON blobs against TypedDict definitions would be useful.
• Distinguishing between Optional items (that can be None) and items that can be omitted altogether feels valuable to me. I don't like the idea of conflating by default.

I don't have a lot of first-hand experience with JSON blobs that have omittable keys, so I don't have especially strong opinions on the semantics for working with them.

[JukkaL]

Okay, here's another proposal:

• By default, keys must exist. Optional[...] as a value type is not special.
• get can be used with arbitrary keys, even those that are declared to always exist. This is for convenience and it doesn't affect safety. The return type for the single-argument version of get is always an optional type, even if the key is supposed to always exist.
• Use OptionalItem[t] for keys that may be missing. This is unrelated to Optional[...].
• It's possible to use OptionalItem[Optional[t]] for an optional item that may be None.
• OptionalItem[...] is only valid if it wraps a typed dict value type. It's an error to use it in any other context.

I chose the name OptionalItem as the name for a few reasons:

• It seems better to reuse the term 'optional' instead of inventing a synonym. This is just a different flavor of optional.
• This declares an optional item -- both the key and the corresponding value may be missing.

More rationale:

• Being an optional item is conceptually orthogonal to accepting a None value, even though the dict.get method somewhat conflates these through the default None value for missing keys.
• Having two independent concepts gives the most flexibility.
• Having two similar concepts is going to be a little confusing. However, OptionalItem is a very searchable name and we can make sure all official documentation about it mentions how it's different from Optional.

Additional notes:

• The join of typed dicts {'x': int} and {'x': int, 'y': str} probably should be {'x': int, 'y': OptionalItem[str]} for convenience. This is not safe, but I doubt that it matters in practice. I can create a separate issue for this with some rationale if we decide to move forward with the rest of this proposal.

[gvanrossum]

Hm, that's pretty verbose, and many people probably don't even know whether their code cares about explicit None values vs. missing keys. How about intentionally conflating the two? The rules would be:

• Items marked up without Optional are mandatory; d[k] is okay and has the stated type; d.get(k) is okay and has an Optional type
• Items marked up with Optional may be None or missing; d[k] is not okay for these; d.get(k) must be used and has an Optional` type
• If you really want a mandatory key with an optional type you can write it as Union[None, T]
• For two-argument d.get(k, default) the return type is a union of None, the type of default, and the nominal type of d[k]

The one use case that this doesn't cover is when you want to allow the key to be missing, but when present you don't want the value to be None. Is that an important use case? If it is, I will withdraw this and then I am okay with the OptionalItem proposal.

[JukkaL]

My gut feeling is that a missing key with the value never being None is not rare. For example, some code I looked at today had code like d.get('x', {}).get('y', 1) which would fail if the return type of get would always include None.

[JukkaL]

Schema evolution or versioning is a typical case where it seems natural to have missing keys with non-Optional values. If we add a new item to a JSON dictionary, but we need to be prepared to accept older objects with a missing key (see above for a discussion about potential use cases), the most likely result seems to be a missing key with a non-optional type.

[gvanrossum]

OK, I withdraw my proposal. IIRC @rowillia also posted some example code that used d.get(k1, {}).get(k2) so apparently this is a common assumption that we should be able to express. And for that case we would just write OptionalItem[...] so that's quite fine. +1!

[davidfstr]

I like the OptionalItem proposal, mainly on the rationale that explicit is better than implicit.