Type annotation erasure at compile time

[ambv]

Problem definition

Current usage of type hints shows the following patterns:

• users often use "Dict", "List", "Set" without adding the required imports from typing; this is especially possible in case of type comments which don't fail at runtime;
• users try to put generics on built-in collections, which again mostly happens in type comments which don't fail at runtime;
• users get confused as to why there is List but there's no Str.

On top of this, I noticed the following tricky typing situation (generalization of actual code at FB):

if TYPE_CHECKING:
    from expensive_unrelated_module import SomeType  # noqa
    from dependency_cycle import CycleType  # noqa

def function(arg: 'SomeType') -> 'CycleType':
    from dependency_cycle import CycleType  # noqa
    return CycleType(arg)

You cannot simply write SomeType or CycleType because it would fail at runtime. So you wrap it in strings. But when you wrap, linters start reporting the imports as unused, or shadowing previous unused imports. So you need to additionally add silencing comments. The resulting code is pretty hideous.

Solution

I'm proposing revisiting the PEP 484 suggestion to make all annotations evaluate to strings at runtime with Python 3.7.

Rationale:

• this lets us seamlessly integrate generics on builtin collections without the need to modify C code to support indexing;
• this provides seamless forward reference support;
• this gives the user seamless access to imports made in the if TYPE_CHECKING: block;
• this accelerates import time and decreases memory usage at runtime.

Details:

• access to an evaluated expression is still possible via typing.get_type_hints();
• the expression is still expected to be syntactically valid and using known names;
• for 3.7 a future import would be required, tentatively called static_annotations (2 characters longer than absolute_imports, 4 characters than with_statement and print_function)
• no other behavioral changes are expected.

Summing up, I think this greatly improves the user experience.

[ambv]

If we get an agreement, I'm going to PEPify it and implement it.

[ilevkivskyi]

I think I am +1 on this, but I would prefer to not make this a default behaviour in 3.7 (at least for one release cycle or as it was done for PEP 479) but rather use a flag or __future__ import. I think there are many people who still rely on annotations being evaluated.

Also I noticed you changed title to "... at parse time", but note that some syntax errors are detected in ast.c i.e. after actual parsing (this is related to Python having very simple parser), so that I would rather "stringify" annotations at a later stage in compile.c (this has a bonus of having them in AST in a form of expression, not string)

[ambv]

I agree it should be in compile.c.

[JukkaL]

I agree that something like this would be better than the current situation.

There are bunch of cases where this wouldn't help as the type is used outside an annotation. Examples (there may be others):

• Alias = <type>
• cast(<type>, e)
• T = TypeVar('T', <type>, ...)
• T = TypeVar('T', bound=<type>)
• T = NewType('T', <type>)
• Non-class-based syntax for named tuples

For these we could continue to use the existing workarounds, such as List[x] or string literal escaping. However, it may be best to deprecate List[x] and friends and recommend exclusively using string literal escaping with the new behavior so that there would no longer be any list[x] vs. List[x] inconsistency.

[ambv]

Yes, for some cases we would need to keep using the current syntax but for most things it could go away. As for aliases, you'd put them in if TYPE_CHECKING:. And note what's possible now:

T: TypeVar
T: TypeVar[T1, T2, ...]
T: BoundTypeVar[SomeType]
T: NewType[<type>]

Solves our problem and looks more elegant.

[JukkaL]

As discussed in python/mypy#2869, generic base classes would also not support this syntax, and there we also can't use string literal escaping. We could continue to write class C(List[int]):, though. String literal escaping also tricky for type aliases -- it's unclear how we'd know whether x = 'list[int]' is supposed to define a type alias or not. So deprecating typing.List and others would perhaps not be possible.

[ilevkivskyi]

@JukkaL

However, it may be best to deprecate List[x] and friends and recommend exclusively using string literal escaping with the new behavior so that there would no longer be any list[x] vs. List[x] inconsistency.

But what about using them in base classes? I think some people still might want to write:

class UserList(List[X]):
    ...
class MyTuple(Tuple[int, str]):
    ...

etc.

---

@ambv
T: TypeVar is interesting, but note that this leaves name T uninitialized, so that class Node(Generic[T]): ... will raise NameError. Maybe we can do something like:

T: TypeVar = (str, int) # for variable with type restrictions
T: TypeVar = Bound[SomeType]
T: NewType = OldType

etc.

[ilevkivskyi]

@JukkaL
Looks like we have a race condition here :-)

[ambv]

I didn't propose to downright deprecate List, and I also just wanted to signal that there are ways in which we can solve TypeVars. Defining new collection types is the outlier situation that would need to exercise the current style. Otherwise we can put most things either in if TYPE_CHECKING or as function/variable annotations.

But before we dive deep into followups from this idea, I'd like to make sure I get the "go ahead" from @gvanrossum first :)

[drhagen]

My understanding was that access to the types at runtime (like reflection) was a large motivator for resolving the annotations rather than leaving them as strings. But the need to wrap some types in strings comes almost entirely from these names being resolved at definition time rather than at runtime. Python could eliminate the need to string wrap forward references while increasing the types available at runtime if it parsed the annotations, but kept them as expressions (probably behind some kind of getter) that could be evaluated at runtime.