diff --git a/docs/source/additional_features.rst b/docs/source/additional_features.rst index 13f9b4cdd160..44a98a4dd266 100644 --- a/docs/source/additional_features.rst +++ b/docs/source/additional_features.rst @@ -1,7 +1,124 @@ Additional features ------------------- -This section discusses various features outside core mypy features. +This section discusses various features that did not fit in naturally in one +of the previous sections. + +.. _function-overloading: + +Function overloading +******************** + +Sometimes the types in a function depend on each other in ways that +can't be captured with a ``Union``. For example, the ``__getitem__`` +(``[]`` bracket indexing) method can take an integer and return a +single item, or take a ``slice`` and return a ``Sequence`` of items. +You might be tempted to annotate it like so: + +.. code-block:: python + + from typing import Sequence, TypeVar, Union + T = TypeVar('T') + + class MyList(Sequence[T]): + def __getitem__(self, index: Union[int, slice]) -> Union[T, Sequence[T]]: + if isinstance(index, int): + ... # Return a T here + elif isinstance(index, slice): + ... # Return a sequence of Ts here + else: + raise TypeError(...) + +But this is too loose, as it implies that when you pass in an ``int`` +you might sometimes get out a single item and sometimes a sequence. +The return type depends on the parameter type in a way that can't be +expressed using a type variable. Instead, we can use `overloading +`_ +to give the same function multiple type annotations (signatures) and +accurately describe the function's behavior. + +.. code-block:: python + + from typing import overload, Sequence, TypeVar, Union + T = TypeVar('T') + + class MyList(Sequence[T]): + + # The @overload definitions are just for the type checker, + # and overwritten by the real implementation below. + @overload + def __getitem__(self, index: int) -> T: + pass # Don't put code here + + # All overloads and the implementation must be adjacent + # in the source file, and overload order may matter: + # when two overloads may overlap, the more specific one + # should come first. + @overload + def __getitem__(self, index: slice) -> Sequence[T]: + pass # Don't put code here + + # The implementation goes last, without @overload. + # It may or may not have type hints; if it does, + # these are checked against the overload definitions + # as well as against the implementation body. + def __getitem__(self, index: Union[int, slice]) -> Union[T, Sequence[T]]: + # This is exactly the same as before. + if isinstance(index, int): + ... # Return a T here + elif isinstance(index, slice): + ... # Return a sequence of Ts here + else: + raise TypeError(...) + +Calls to overloaded functions are type checked against the variants, +not against the implementation. A call like ``my_list[5]`` would have +type ``T``, not ``Union[T, Sequence[T]]`` because it matches the +first overloaded definition, and ignores the type annotations on the +implementation of ``__getitem__``. The code in the body of the +definition of ``__getitem__`` is checked against the annotations on +the corresponding declaration. In this case the body is checked +with ``index: Union[int, slice]`` and a return type +``Union[T, Sequence[T]]``. If there are no annotations on the +corresponding definition, then code in the function body is not type +checked. + +The annotations on the function body must be compatible with the +types given for the overloaded variants listed above it. The type +checker will verify that all the types for the overloaded variants +are compatible with the types given for the implementation. In this +case it checks that the parameter type ``int`` and the return type +``T`` are compatible with ``Union[int, slice]`` and +``Union[T, Sequence[T]]`` for the first variant. For the second +variant it verifies that the parameter type ``slice`` and the return +type ``Sequence[T]`` are compatible with ``Union[int, slice]`` and +``Union[T, Sequence[T]]``. + +Overloaded function variants are still ordinary Python functions and +they still define a single runtime object. There is no automatic +dispatch happening, and you must manually handle the different types +in the implementation (usually with :func:`isinstance` checks, as +shown in the example). + +The overload variants must be adjacent in the code. This makes code +clearer, as you don't have to hunt for overload variants across the +file. + +Overloads in stub files are exactly the same, except there is no +implementation. + +.. note:: + + As generic type variables are erased at runtime when constructing + instances of generic types, an overloaded function cannot have + variants that only differ in a generic type argument, + e.g. ``List[int]`` and ``List[str]``. + +.. note:: + + If you just need to constrain a type variable to certain types or + subtypes, you can use a :ref:`value restriction + `. .. _attrs_package: @@ -17,6 +134,7 @@ Type annotations can be added as follows: .. code-block:: python import attr + @attr.s class A: one: int = attr.ib() # Variable annotation (Python 3.6+) @@ -28,6 +146,7 @@ If you're using ``auto_attribs=True`` you must use variable annotations. .. code-block:: python import attr + @attr.s(auto_attribs=True) class A: one: int @@ -43,6 +162,7 @@ That enables this to work: import attr from typing import Dict + @attr.s(auto_attribs=True) class A: one: int = attr.ib(8) @@ -175,7 +295,7 @@ Caching with mypy daemon ======================== You can also use remote caching with the :ref:`mypy daemon `. -The remote cache will significantly speed up the the first ``dmypy check`` +The remote cache will significantly speed up the first ``dmypy check`` run after starting or restarting the daemon. The mypy daemon requires extra fine-grained dependency data in @@ -231,3 +351,129 @@ at least if your codebase is hundreds of thousands of lines or more: mypy build to create the cache data, as repeatedly updating cache data incrementally could result in drift over a long time period (due to a mypy caching issue, perhaps). + +.. _extended_callable: + +Extended Callable types +*********************** + +As an experimental mypy extension, you can specify ``Callable`` types +that support keyword arguments, optional arguments, and more. When +you specify the arguments of a Callable, you can choose to supply just +the type of a nameless positional argument, or an "argument specifier" +representing a more complicated form of argument. This allows one to +more closely emulate the full range of possibilities given by the +``def`` statement in Python. + +As an example, here's a complicated function definition and the +corresponding ``Callable``: + +.. code-block:: python + + from typing import Callable + from mypy_extensions import (Arg, DefaultArg, NamedArg, + DefaultNamedArg, VarArg, KwArg) + + def func(__a: int, # This convention is for nameless arguments + b: int, + c: int = 0, + *args: int, + d: int, + e: int = 0, + **kwargs: int) -> int: + ... + + F = Callable[[int, # Or Arg(int) + Arg(int, 'b'), + DefaultArg(int, 'c'), + VarArg(int), + NamedArg(int, 'd'), + DefaultNamedArg(int, 'e'), + KwArg(int)], + int] + + f: F = func + +Argument specifiers are special function calls that can specify the +following aspects of an argument: + +- its type (the only thing that the basic format supports) + +- its name (if it has one) + +- whether it may be omitted + +- whether it may or must be passed using a keyword + +- whether it is a ``*args`` argument (representing the remaining + positional arguments) + +- whether it is a ``**kwargs`` argument (representing the remaining + keyword arguments) + +The following functions are available in ``mypy_extensions`` for this +purpose: + +.. code-block:: python + + def Arg(type=Any, name=None): + # A normal, mandatory, positional argument. + # If the name is specified it may be passed as a keyword. + + def DefaultArg(type=Any, name=None): + # An optional positional argument (i.e. with a default value). + # If the name is specified it may be passed as a keyword. + + def NamedArg(type=Any, name=None): + # A mandatory keyword-only argument. + + def DefaultNamedArg(type=Any, name=None): + # An optional keyword-only argument (i.e. with a default value). + + def VarArg(type=Any): + # A *args-style variadic positional argument. + # A single VarArg() specifier represents all remaining + # positional arguments. + + def KwArg(type=Any): + # A **kwargs-style variadic keyword argument. + # A single KwArg() specifier represents all remaining + # keyword arguments. + +In all cases, the ``type`` argument defaults to ``Any``, and if the +``name`` argument is omitted the argument has no name (the name is +required for ``NamedArg`` and ``DefaultNamedArg``). A basic +``Callable`` such as + +.. code-block:: python + + MyFunc = Callable[[int, str, int], float] + +is equivalent to the following: + +.. code-block:: python + + MyFunc = Callable[[Arg(int), Arg(str), Arg(int)], float] + +A ``Callable`` with unspecified argument types, such as + +.. code-block:: python + + MyOtherFunc = Callable[..., int] + +is (roughly) equivalent to + +.. code-block:: python + + MyOtherFunc = Callable[[VarArg(), KwArg()], int] + +.. note:: + + This feature is experimental. Details of the implementation may + change and there may be unknown limitations. **IMPORTANT:** + Each of the functions above currently just returns its ``type`` + argument, so the information contained in the argument specifiers + is not available at runtime. This limitation is necessary for + backwards compatibility with the existing ``typing.py`` module as + present in the Python 3.5+ standard library and distributed via + PyPI. diff --git a/docs/source/casts.rst b/docs/source/casts.rst index 900ee0ce4220..741513744a3e 100644 --- a/docs/source/casts.rst +++ b/docs/source/casts.rst @@ -1,7 +1,7 @@ .. _casts: -Casts -===== +Casts and type assertions +========================= Mypy supports type casts that are usually used to coerce a statically typed value to a subtype. Unlike languages such as Java or C#, @@ -13,17 +13,27 @@ cast: from typing import cast, List - o = [1] # type: object + o: object = [1] x = cast(List[int], o) # OK y = cast(List[str], o) # OK (cast performs no actual runtime check) To support runtime checking of casts such as the above, we'd have to check the types of all list items, which would be very inefficient for large lists. -Use assertions if you want to -perform an actual runtime check. Casts are used to silence spurious +Casts are used to silence spurious type checker warnings and give the type checker a little help when it can't quite understand what is going on. +.. note:: + + You can use an assertion if you want to perform an actual runtime check: + + .. code-block:: python + + def foo(o: object) -> None: + print(o + 5) # Error: can't add 'object' and 'int' + assert isinstance(o, int) + print(o + 5) # OK: type of 'o' is 'int' here + You don't need a cast for expressions with type ``Any``, or when assigning to a variable with type ``Any``, as was explained earlier. You can also use ``Any`` as the cast target type -- this lets you perform @@ -34,6 +44,6 @@ any operations on the result. For example: from typing import cast, Any x = 1 - x + 'x' # Type check error + x.whatever() # Type check error y = cast(Any, x) - y + 'x' # Type check OK (runtime error) + y.whatever() # Type check OK (runtime error) diff --git a/docs/source/class_basics.rst b/docs/source/class_basics.rst index 4bb8018c06cd..fc830146633a 100644 --- a/docs/source/class_basics.rst +++ b/docs/source/class_basics.rst @@ -1,6 +1,10 @@ Class basics ============ +This section will help get you started annotating your +classes. Built-in classes such as ``int`` also follow these same +rules. + Instance and class attributes ***************************** @@ -13,29 +17,42 @@ initialized within the class. Mypy infers the types of attributes: class A: def __init__(self, x: int) -> None: - self.x = x # Attribute x of type int + self.x = x # Aha, attribute 'x' of type 'int' a = A(1) - a.x = 2 # OK - a.y = 3 # Error: A has no attribute y + a.x = 2 # OK! + a.y = 3 # Error: 'A' has no attribute 'y' This is a bit like each class having an implicitly defined ``__slots__`` attribute. This is only enforced during type checking and not when your program is running. You can declare types of variables in the class body explicitly using -a type comment: +a type annotation: .. code-block:: python class A: - x = None # type: List[int] # Declare attribute x of type List[int] + x: List[int] # Declare attribute 'x' of type List[int] a = A() a.x = [1] # OK -As in Python, a variable defined in the class body can used as a class -or an instance variable. +As in Python generally, a variable defined in the class body can used +as a class or an instance variable. + +Type comments work as well, if you need to support Python versions earlier +than 3.6: + +.. code-block:: python + + class A: + x = None # type: List[int] # Declare attribute 'x' of type List[int] + +Note that attribute definitions in the class body that use a type comment +are special: a ``None`` value is valid as the initializer, even though +the declared type is not optional. This should be used sparingly, as this can +result in ``None``-related runtime errors that mypy can't detect. Similarly, you can give explicit types to instance variables defined in a method: @@ -44,10 +61,10 @@ in a method: class A: def __init__(self) -> None: - self.x = [] # type: List[int] + self.x: List[int] = [] def f(self) -> None: - self.y = 0 # type: Any + self.y: Any = 0 You can only define an instance variable within a method if you assign to it explicitly using ``self``: @@ -56,9 +73,9 @@ to it explicitly using ``self``: class A: def __init__(self) -> None: - self.y = 1 # Define y + self.y = 1 # Define 'y' a = self - a.x = 1 # Error: x not defined + a.x = 1 # Error: 'x' not defined Overriding statically typed methods *********************************** @@ -73,7 +90,7 @@ override has a compatible signature: ... class B(A): - def f(self, x: str) -> None: # Error: type of x incompatible + def f(self, x: str) -> None: # Error: type of 'x' incompatible ... class C(A): @@ -88,16 +105,18 @@ override has a compatible signature: You can also vary return types **covariantly** in overriding. For example, you could override the return type ``object`` with a subtype - such as ``int``. + such as ``int``. Similarly, you can vary argument types + **contravariantly** -- subclasses can have more general argument types. You can also override a statically typed method with a dynamically typed one. This allows dynamically typed code to override methods defined in library classes without worrying about their type signatures. -There is no runtime enforcement that the method override returns a -value that is compatible with the original return type, since -annotations have no effect at runtime: +As always, relying on dynamically typed code can be unsafe. There is no +runtime enforcement that the method override returns a value that is +compatible with the original return type, since annotations have no +effect at runtime: .. code-block:: python @@ -106,13 +125,8 @@ annotations have no effect at runtime: return x + 1 class B(A): - def inc(self, x): # Override, dynamically typed - return 'hello' - - b = B() - print(b.inc(1)) # hello - a = b # type: A - print(a.inc(1)) # hello + def inc(self, x): # Override, dynamically typed + return 'hello' # Incompatible with 'A', but no mypy error Abstract base classes and multiple inheritance ********************************************** @@ -139,7 +153,7 @@ by a subclass. You can define abstract base classes using the def bar(self) -> str: return 'x' - a = A() # Error: A is abstract + a = A() # Error: 'A' is abstract b = B() # OK Note that mypy performs checking for unimplemented abstract methods @@ -153,414 +167,3 @@ including an abstract method defined in an abstract base class. You can implement an abstract property using either a normal property or an instance variable. - -.. _protocol-types: - -Protocols and structural subtyping -********************************** - -Mypy supports two ways of deciding whether two classes are compatible -as types: nominal subtyping and structural subtyping. *Nominal* -subtyping is strictly based on the class hierarchy. If class ``D`` -inherits class ``C``, it's also a subtype of ``C``, and instances of -``D`` can be used when ``C`` instances are expected. This form of -subtyping is used by default in mypy, since it's easy to understand -and produces clear and concise error messages, and since it matches -how the native ``isinstance()`` check works -- based on class -hierarchy. *Structural* subtyping can also be useful. Class ``D`` is -a structural subtype of class ``C`` if the former has all attributes -and methods of the latter, and with compatible types. - -Structural subtyping can be seen as a static equivalent of duck -typing, which is well known to Python programmers. Mypy provides -support for structural subtyping via protocol classes described -below. See `PEP 544 `_ for -the detailed specification of protocols and structural subtyping in -Python. - -.. _predefined_protocols: - -Predefined protocols -******************** - -The ``typing`` module defines various protocol classes that correspond -to common Python protocols, such as ``Iterable[T]``. If a class -defines a suitable ``__iter__`` method, mypy understands that it -implements the iterable protocol and is compatible with ``Iterable[T]``. -For example, ``IntList`` below is iterable, over ``int`` values: - -.. code-block:: python - - from typing import Iterator, Iterable, Optional - - class IntList: - def __init__(self, value: int, next: Optional[IntList]) -> None: - self.value = value - self.next = next - - def __iter__(self) -> Iterator[int]: - current = self - while current: - yield current.value - current = current.next - - def print_numbered(items: Iterable[int]) -> None: - for n, x in enumerate(items): - print(n + 1, x) - - x = IntList(3, IntList(5, None)) - print_numbered(x) # OK - print_numbered([4, 5]) # Also OK - -The subsections below introduce all built-in protocols defined in -``typing`` and the signatures of the corresponding methods you need to define -to implement each protocol (the signatures can be left out, as always, but mypy -won't type check unannotated methods). - -Iteration protocols -................... - -The iteration protocols are useful in many contexts. For example, they allow -iteration of objects in for loops. - -``Iterable[T]`` ---------------- - -The :ref:`example above ` has a simple implementation of an -``__iter__`` method. - -.. code-block:: python - - def __iter__(self) -> Iterator[T] - -``Iterator[T]`` ---------------- - -.. code-block:: python - - def __next__(self) -> T - def __iter__(self) -> Iterator[T] - -Collection protocols -.................... - -Many of these are implemented by built-in container types such as -``list`` and ``dict``, and these are also useful for user-defined -collection objects. - -``Sized`` ---------- - -This is a type for objects that support ``len(x)``. - -.. code-block:: python - - def __len__(self) -> int - -``Container[T]`` ----------------- - -This is a type for objects that support the ``in`` operator. - -.. code-block:: python - - def __contains__(self, x: object) -> bool - -``Collection[T]`` ------------------ - -.. code-block:: python - - def __len__(self) -> int - def __iter__(self) -> Iterator[T] - def __contains__(self, x: object) -> bool - -One-off protocols -................. - -These protocols are typically only useful with a single standard -library function or class. - -``Reversible[T]`` ------------------ - -This is a type for objects that support ``reversed(x)``. - -.. code-block:: python - - def __reversed__(self) -> Iterator[T] - -``SupportsAbs[T]`` ------------------- - -This is a type for objects that support ``abs(x)``. ``T`` is the type of -value returned by ``abs(x)``. - -.. code-block:: python - - def __abs__(self) -> T - -``SupportsBytes`` ------------------ - -This is a type for objects that support ``bytes(x)``. - -.. code-block:: python - - def __bytes__(self) -> bytes - -.. _supports-int-etc: - -``SupportsComplex`` -------------------- - -This is a type for objects that support ``complex(x)``. Note that no arithmetic operations -are supported. - -.. code-block:: python - - def __complex__(self) -> complex - -``SupportsFloat`` ------------------ - -This is a type for objects that support ``float(x)``. Note that no arithmetic operations -are supported. - -.. code-block:: python - - def __float__(self) -> float - -``SupportsInt`` ---------------- - -This is a type for objects that support ``int(x)``. Note that no arithmetic operations -are supported. - -.. code-block:: python - - def __int__(self) -> int - -``SupportsRound[T]`` --------------------- - -This is a type for objects that support ``round(x)``. - -.. code-block:: python - - def __round__(self) -> T - -Async protocols -............... - -These protocols can be useful in async code. - -``Awaitable[T]`` ----------------- - -.. code-block:: python - - def __await__(self) -> Generator[Any, None, T] - -``AsyncIterable[T]`` --------------------- - -.. code-block:: python - - def __aiter__(self) -> AsyncIterator[T] - -``AsyncIterator[T]`` --------------------- - -.. code-block:: python - - def __anext__(self) -> Awaitable[T] - def __aiter__(self) -> AsyncIterator[T] - -Context manager protocols -......................... - -There are two protocols for context managers -- one for regular context -managers and one for async ones. These allow defining objects that can -be used in ``with`` and ``async with`` statements. - -``ContextManager[T]`` ---------------------- - -.. code-block:: python - - def __enter__(self) -> T - def __exit__(self, - exc_type: Optional[Type[BaseException]], - exc_value: Optional[BaseException], - traceback: Optional[TracebackType]) -> Optional[bool] - -``AsyncContextManager[T]`` --------------------------- - -.. code-block:: python - - def __aenter__(self) -> Awaitable[T] - def __aexit__(self, - exc_type: Optional[Type[BaseException]], - exc_value: Optional[BaseException], - traceback: Optional[TracebackType]) -> Awaitable[Optional[bool]] - -Simple user-defined protocols -***************************** - -You can define your own protocol class by inheriting the special -``typing_extensions.Protocol`` class: - -.. code-block:: python - - from typing import Iterable - from typing_extensions import Protocol - - class SupportsClose(Protocol): - def close(self) -> None: - ... # Explicit '...' - - class Resource: # No SupportsClose base class! - # ... some methods ... - - def close(self) -> None: - self.resource.release() - - def close_all(items: Iterable[SupportsClose]) -> None: - for item in items: - item.close() - - close_all([Resource(), open('some/file')]) # Okay! - -``Resource`` is a subtype of the ``SupportClose`` protocol since it defines -a compatible ``close`` method. Regular file objects returned by ``open()`` are -similarly compatible with the protocol, as they support ``close()``. - -.. note:: - - The ``Protocol`` base class is currently provided in the ``typing_extensions`` - package. Once structural subtyping is mature and - `PEP 544 `_ has been accepted, - ``Protocol`` will be included in the ``typing`` module. - -Defining subprotocols and subclassing protocols -*********************************************** - -You can also define subprotocols. Existing protocols can be extended -and merged using multiple inheritance. Example: - -.. code-block:: python - - # ... continuing from the previous example - - class SupportsRead(Protocol): - def read(self, amount: int) -> bytes: ... - - class TaggedReadableResource(SupportsClose, SupportsRead, Protocol): - label: str - - class AdvancedResource(Resource): - def __init__(self, label: str) -> None: - self.label = label - - def read(self, amount: int) -> bytes: - # some implementation - ... - - resource: TaggedReadableResource - resource = AdvancedResource('handle with care') # OK - -Note that inheriting from an existing protocol does not automatically -turn the subclass into a protocol -- it just creates a regular -(non-protocol) class or ABC that implements the given protocol (or -protocols). The ``typing_extensions.Protocol`` base class must always -be explicitly present if you are defining a protocol: - -.. code-block:: python - - class NewProtocol(SupportsClose): # This is NOT a protocol - new_attr: int - - class Concrete: - new_attr: int = 0 - - def close(self) -> None: - ... - - # Error: nominal subtyping used by default - x: NewProtocol = Concrete() # Error! - -You can also include default implementations of methods in -protocols. If you explicitly subclass these protocols you can inherit -these default implementations. Explicitly including a protocol as a -base class is also a way of documenting that your class implements a -particular protocol, and it forces mypy to verify that your class -implementation is actually compatible with the protocol. - -.. note:: - - You can use Python 3.6 variable annotations (`PEP 526 - `_) - to declare protocol attributes. On Python 2.7 and earlier Python 3 - versions you can use type comments and properties. - -Recursive protocols -******************* - -Protocols can be recursive (self-referential) and mutually -recursive. This is useful for declaring abstract recursive collections -such as trees and linked lists: - -.. code-block:: python - - from typing import TypeVar, Optional - from typing_extensions import Protocol - - class TreeLike(Protocol): - value: int - - @property - def left(self) -> Optional['TreeLike']: ... - - @property - def right(self) -> Optional['TreeLike']: ... - - class SimpleTree: - def __init__(self, value: int) -> None: - self.value = value - self.left: Optional['SimpleTree'] = None - self.right: Optional['SimpleTree'] = None - - root = SimpleTree(0) # type: TreeLike # OK - -Using ``isinstance()`` with protocols -************************************* - -You can use a protocol class with ``isinstance()`` if you decorate it -with the ``typing_extensions.runtime`` class decorator. The decorator -adds support for basic runtime structural checks: - -.. code-block:: python - - from typing_extensions import Protocol, runtime - - @runtime - class Portable(Protocol): - handles: int - - class Mug: - def __init__(self) -> None: - self.handles = 1 - - mug = Mug() - if isinstance(mug, Portable): - use(mug.handles) # Works statically and at runtime - -``isinstance()`` also works with the :ref:`predefined protocols ` -in ``typing`` such as ``Iterable``. - -.. note:: - ``isinstance()`` with protocols is not completely safe at runtime. - For example, signatures of methods are not checked. The runtime - implementation only checks that all protocol members are defined. diff --git a/docs/source/command_line.rst b/docs/source/command_line.rst index ba198a1d7900..d4dfad6c28ad 100644 --- a/docs/source/command_line.rst +++ b/docs/source/command_line.rst @@ -393,7 +393,7 @@ Here are some more useful flags: Otherwise, use ``--python-executable``. - ``--platform PLATFORM`` will make mypy typecheck your code as if it were - run under the the given operating system. Without this option, mypy will + run under the given operating system. Without this option, mypy will default to using whatever operating system you are currently using. See :ref:`version_and_platform_checks` for more about this feature. diff --git a/docs/source/common_issues.rst b/docs/source/common_issues.rst index 1f0a3ead60ad..253f4fc630ee 100644 --- a/docs/source/common_issues.rst +++ b/docs/source/common_issues.rst @@ -1,13 +1,13 @@ .. _common_issues: -Common issues -============= +Common issues and solutions +=========================== This section has examples of cases when you need to update your code to use static typing, and ideas for working around issues if mypy doesn't work as expected. Statically typed code is often identical to -normal Python code, but sometimes you need to do things slightly -differently. +normal Python code (except for type annotations), but sometimes you need +to do things slightly differently. Can't install mypy using pip ---------------------------- @@ -131,16 +131,44 @@ The second line is now fine, since the ignore comment causes the name if we did have a stub available for ``frobnicate`` then mypy would ignore the ``# type: ignore`` comment and typecheck the stub as usual. +Another option is to explicitly annotate values with type ``Any`` -- +mypy will let you perform arbitrary operations on ``Any`` +values. Sometimes there is no more precise type you can use for a +particular value, especially if you use dynamic Python features +such as ``__getattr__``: + +.. code-block:: python + + class Wrapper: + ... + def __getattr__(self, a: str) -> Any: + return getattr(self._wrapped, a) + +Finally, you can create a stub file (``.pyi``) for a file that +generates spurious errors. Mypy will only look at the stub file +and ignore the implementation, since stub files take precedence +over ``.py`` files. Unexpected errors about 'None' and/or 'Optional' types ------------------------------------------------------ Starting from mypy 0.600, mypy uses :ref:`strict optional checking ` by default, -and ``None`` is not compatible with non-optional types. It's -easy to switch back to the older behavior where ``None`` was +and the ``None`` value is not compatible with non-optional types. +It's easy to switch back to the older behavior where ``None`` was compatible with arbitrary types (see :ref:`no_strict_optional`). +You can also fall back to this behavior if strict optional +checking would require a large number of ``assert foo is not None`` +checks to be inserted, and you want to minimize the number +of code changes required to get a clean mypy run. + +Mypy runs are slow +------------------ +If your mypy runs feel slow, you should probably use the :ref:`mypy +daemon `, which can speed up incremental mypy runtimes by +a factor of 10 or more. :ref:`Remote caching ` can +make cold mypy runs several times faster. Types of empty collections -------------------------- @@ -150,7 +178,7 @@ dict to a new variable, as mentioned earlier: .. code-block:: python - a = [] # type: List[int] + a: List[int] = [] Without the annotation mypy can't always figure out the precise type of ``a``. @@ -249,48 +277,6 @@ Possible strategies in such situations are: return x[0] f_good(new_lst) # OK -Covariant subtyping of mutable protocol members is rejected ------------------------------------------------------------ - -Mypy rejects this because this is potentially unsafe. -Consider this example: - -.. code-block:: python - - from typing_extensions import Protocol - - class P(Protocol): - x: float - - def fun(arg: P) -> None: - arg.x = 3.14 - - class C: - x = 42 - c = C() - fun(c) # This is not safe - c.x << 5 # Since this will fail! - -To work around this problem consider whether "mutating" is actually part -of a protocol. If not, then one can use a ``@property`` in -the protocol definition: - -.. code-block:: python - - from typing_extensions import Protocol - - class P(Protocol): - @property - def x(self) -> float: - pass - - def fun(arg: P) -> None: - ... - - class C: - x = 42 - fun(C()) # OK - Declaring a supertype as variable type -------------------------------------- @@ -533,6 +519,48 @@ put the linter comment *after* the type comment: a = some_complex_thing() # type: ignore # noqa +Covariant subtyping of mutable protocol members is rejected +----------------------------------------------------------- + +Mypy rejects this because this is potentially unsafe. +Consider this example: + +.. code-block:: python + + from typing_extensions import Protocol + + class P(Protocol): + x: float + + def fun(arg: P) -> None: + arg.x = 3.14 + + class C: + x = 42 + c = C() + fun(c) # This is not safe + c.x << 5 # Since this will fail! + +To work around this problem consider whether "mutating" is actually part +of a protocol. If not, then one can use a ``@property`` in +the protocol definition: + +.. code-block:: python + + from typing_extensions import Protocol + + class P(Protocol): + @property + def x(self) -> float: + pass + + def fun(arg: P) -> None: + ... + + class C: + x = 42 + fun(C()) # OK + Dealing with conflicting names ------------------------------ diff --git a/docs/source/dynamic_typing.rst b/docs/source/dynamic_typing.rst index ba76442eb0f5..50b00f5e033f 100644 --- a/docs/source/dynamic_typing.rst +++ b/docs/source/dynamic_typing.rst @@ -15,10 +15,10 @@ dynamically typed by defining it explicitly with the type ``Any``: from typing import Any - s = 1 # Statically typed (type int) - d = 1 # type: Any # Dynamically typed (type Any) - s = 'x' # Type check error - d = 'x' # OK + s = 1 # Statically typed (type int) + d: Any = 1 # Dynamically typed (type Any) + s = 'x' # Type check error + d = 'x' # OK Operations on Any values ------------------------ diff --git a/docs/source/faq.rst b/docs/source/faq.rst index 512d922879a5..995dfc3c9493 100644 --- a/docs/source/faq.rst +++ b/docs/source/faq.rst @@ -6,7 +6,8 @@ Why have both dynamic and static typing? Dynamic typing can be flexible, powerful, convenient and easy. But it's not always the best approach; there are good reasons why many -developers choose to use statically typed languages. +developers choose to use statically typed languages or static typing +for Python. Here are some potential benefits of mypy-style static typing: @@ -54,26 +55,26 @@ may be useful: - Multiple developers are working on the same code. -- Running tests takes a lot of time or work (type checking may help - you find errors early in development, reducing the number of testing - iterations). +- Running tests takes a lot of time or work (type checking helps + you find errors quickly early in development, reducing the number of + testing iterations). - Some project members (devs or management) don't like dynamic typing, but others prefer dynamic typing and Python syntax. Mypy could be a solution that everybody finds easy to accept. - You want to future-proof your project even if currently none of the - above really apply. + above really apply. The earlier you start, the easier it will be to + adopt static typing. Can I use mypy to type check my existing Python code? ***************************************************** -It depends. Compatibility is pretty good, but some Python features are -not yet implemented or fully supported. The ultimate goal is to make -using mypy practical for most Python code. Code that uses complex +Mypy supports most Python features and idioms, and many large Python +projects are using mypy successfully. Code that uses complex introspection or metaprogramming may be impractical to type check, but it should still be possible to use static typing in other parts of a -program. +codebase that are less dynamic. Will static typing make my programs run faster? *********************************************** @@ -82,10 +83,7 @@ Mypy only does static type checking and it does not improve performance. It has a minimal performance impact. In the future, there could be other tools that can compile statically typed mypy code to C modules or to efficient JVM bytecode, for example, but this is outside -the scope of the mypy project. It may also be possible to modify -existing Python VMs to take advantage of static type information, but -whether this is feasible is still unknown. This is nontrivial since -the runtime types do not necessarily correspond to the static types. +the scope of the mypy project. How do I type check my Python 2 code? ************************************* @@ -101,13 +99,14 @@ Is mypy free? Yes. Mypy is free software, and it can also be used for commercial and proprietary projects. Mypy is available under the MIT license. -Can I use structural subtyping? -******************************* +Can I use duck typing with mypy? +******************************** Mypy provides support for both `nominal subtyping `_ and `structural subtyping `_. +Structural subtyping can be thought of as "static duck typing". Some argue that structural subtyping is better suited for languages with duck typing such as Python. Mypy however primarily uses nominal subtyping, leaving structural subtyping mostly opt-in (except for built-in protocols @@ -138,11 +137,12 @@ subtyping see :ref:`protocol-types` and I like Python and I have no need for static typing ************************************************** -That wasn't really a question, was it? Mypy is not aimed at replacing -Python. The goal is to give more options for Python programmers, to +The aim of mypy is not to convince everybody to write statically typed +Python -- static typing is entirely optional, now and in the +future. The goal is to give more options for Python programmers, to make Python a more competitive alternative to other statically typed -languages in large projects, to improve programmer productivity and to -improve software quality. +languages in large projects, to improve programmer productivity, and +to improve software quality. How are mypy programs different from normal Python? *************************************************** @@ -156,16 +156,11 @@ supported by mypy, but this is gradually improving. The obvious difference is the availability of static type checking. The section :ref:`common_issues` mentions some modifications to Python code that may be required to make code type -check without errors. Also, your code must make attributes explicit and -use a explicit protocol representation. For example, you may want to -subclass an Abstract Base Class such as ``typing.Iterable``. +check without errors. Also, your code must make attributes explicit. -Mypy will support modular, efficient type checking, and this seems to +Mypy supports modular, efficient type checking, and this seems to rule out type checking some language features, such as arbitrary -runtime addition of methods. However, it is likely that many of these -features will be supported in a restricted form (for example, runtime -modification is only supported for classes or methods registered as -dynamic or 'patchable'). +monkey patching of methods. How is mypy different from Cython? ********************************** @@ -205,53 +200,6 @@ the following aspects, among others: Python semantics, and mypy does not deal with accessing C library functionality. -How is mypy different from Nuitka? -********************************** - -`Nuitka `_ is a static compiler that can translate -Python programs to C++. Nuitka integrates with the CPython -runtime. Nuitka has additional future goals, such as using type -inference and whole-program analysis to further speed up code. Here -are some differences: - -- Nuitka is primarily focused on speeding up Python code. Mypy focuses - on static type checking and facilitating better tools. - -- Whole-program analysis tends to be slow and scale poorly to large or - complex programs. It is still unclear if Nuitka can solve these - issues. Mypy does not use whole-program analysis and will support - modular type checking (though this has not been implemented yet). - -How is mypy different from RPython or Shed Skin? -************************************************ - -`RPython `_ and `Shed -Skin `_ are basically statically -typed subsets of Python. Mypy does the following important things -differently: - -- RPython is primarily designed for implementing virtual machines; - mypy is a general-purpose tool. - -- Mypy supports both static and dynamic typing. Dynamically typed and - statically typed code can be freely mixed and can interact - seamlessly. - -- Mypy aims to support (in the future) fast and modular type - checking. Both RPython and Shed Skin use whole-program type - inference which is very slow, does not scale well to large programs - and often produces confusing error messages. Mypy can support - modularity since it only uses local type inference; static type - checking depends on having type annotations for functions - signatures. - -- Mypy will support introspection, dynamic loading of code and many - other dynamic language features (though using these may make static - typing less effective). RPython and Shed Skin only support a - restricted Python subset without several of these features. - -- Mypy supports user-defined generic types. - Mypy is a cool project. Can I help? *********************************** diff --git a/docs/source/function_overloading.rst b/docs/source/function_overloading.rst deleted file mode 100644 index 5e7d2c5b1a82..000000000000 --- a/docs/source/function_overloading.rst +++ /dev/null @@ -1,115 +0,0 @@ -.. _function-overloading: - -Function Overloading -==================== - -Sometimes the types in a function depend on each other in ways that -can't be captured with a ``Union``. For example, the ``__getitem__`` -(``[]`` bracket indexing) method can take an integer and return a -single item, or take a ``slice`` and return a ``Sequence`` of items. -You might be tempted to annotate it like so: - -.. code-block:: python - - from typing import Sequence, TypeVar, Union - T = TypeVar('T') - - class MyList(Sequence[T]): - def __getitem__(self, index: Union[int, slice]) -> Union[T, Sequence[T]]: - if isinstance(index, int): - ... # Return a T here - elif isinstance(index, slice): - ... # Return a sequence of Ts here - else: - raise TypeError(...) - -But this is too loose, as it implies that when you pass in an ``int`` -you might sometimes get out a single item and sometimes a sequence. -The return type depends on the parameter type in a way that can't be -expressed using a type variable. Instead, we can use `overloading -`_ -to give the same function multiple type annotations (signatures) and -accurately describe the function's behavior. - -.. code-block:: python - - from typing import overload, Sequence, TypeVar, Union - T = TypeVar('T') - - class MyList(Sequence[T]): - - # The @overload definitions are just for the type checker, - # and overwritten by the real implementation below. - @overload - def __getitem__(self, index: int) -> T: - pass # Don't put code here - - # All overloads and the implementation must be adjacent - # in the source file, and overload order may matter: - # when two overloads may overlap, the more specific one - # should come first. - @overload - def __getitem__(self, index: slice) -> Sequence[T]: - pass # Don't put code here - - # The implementation goes last, without @overload. - # It may or may not have type hints; if it does, - # these are checked against the overload definitions - # as well as against the implementation body. - def __getitem__(self, index: Union[int, slice]) -> Union[T, Sequence[T]]: - # This is exactly the same as before. - if isinstance(index, int): - ... # Return a T here - elif isinstance(index, slice): - ... # Return a sequence of Ts here - else: - raise TypeError(...) - -Calls to overloaded functions are type checked against the variants, -not against the implementation. A call like ``my_list[5]`` would have -type ``T``, not ``Union[T, Sequence[T]]`` because it matches the -first overloaded definition, and ignores the type annotations on the -implementation of ``__getitem__``. The code in the body of the -definition of ``__getitem__`` is checked against the annotations on -the the corresponding declaration. In this case the body is checked -with ``index: Union[int, slice]`` and a return type -``Union[T, Sequence[T]]``. If there are no annotations on the -corresponding definition, then code in the function body is not type -checked. - -The annotations on the function body must be compatible with the -types given for the overloaded variants listed above it. The type -checker will verify that all the types listed the overloaded variants -are compatible with the types given for the implementation. In this -case it checks that the parameter type ``int`` and the return type -``T`` are compatible with ``Union[int, slice]`` and -``Union[T, Sequence[T]]`` for the first variant. For the second -variant it verifies that the parameter type ``slice`` are the return -type ``Sequence[T]`` are compatible with ``Union[int, slice]`` and -``Union[T, Sequence[T]]``. - -Overloaded function variants are still ordinary Python functions and -they still define a single runtime object. There is no automatic -dispatch happening, and you must manually handle the different types -in the implementation (usually with :func:`isinstance` checks, as -shown in the example). - -The overload variants must be adjacent in the code. This makes code -clearer, as you don't have to hunt for overload variants across the -file. - -Overloads in stub files are exactly the same, except there is no -implementation. - -.. note:: - - As generic type variables are erased at runtime when constructing - instances of generic types, an overloaded function cannot have - variants that only differ in a generic type argument, - e.g. ``List[int]`` and ``List[str]``. - -.. note:: - - If you just need to constrain a type variable to certain types or - subtypes, you can use a :ref:`value restriction - `. diff --git a/docs/source/generics.rst b/docs/source/generics.rst index 4b97a10d1f81..0d3023bd1e5a 100644 --- a/docs/source/generics.rst +++ b/docs/source/generics.rst @@ -1,6 +1,11 @@ Generics ======== +This section explains how you can define your own generic classes that take +one or more type parameters, similar to built-in types such as ``List[X]``. +User-defined generics are a moderately advanced feature and you can get far +without ever using them -- feel free to skip this section and come back later. + .. _generic-classes: Defining generic classes @@ -23,7 +28,7 @@ generic class that represents a stack: class Stack(Generic[T]): def __init__(self) -> None: # Create an empty list with items of type T - self.items = [] # type: List[T] + self.items: List[T] = [] def push(self, item: T) -> None: self.items.append(item) @@ -644,3 +649,76 @@ Generic protocols can also be recursive. Example: ... # implementation omitted result = last(L()) # Inferred type of 'result' is 'int' + +Generic type aliases +******************** + +Type aliases can be generic. In this case they can be used in two ways: +Subscripted aliases are equivalent to original types with substituted type +variables, so the number of type arguments must match the number of free type variables +in the generic type alias. Unsubscripted aliases are treated as original types with free +variables replaced with ``Any``. Examples (following `PEP 484 +`_): + +.. code-block:: python + + from typing import TypeVar, Iterable, Tuple, Union, Callable + + S = TypeVar('S') + + TInt = Tuple[int, S] + UInt = Union[S, int] + CBack = Callable[..., S] + + def response(query: str) -> UInt[str]: # Same as Union[str, int] + ... + def activate(cb: CBack[S]) -> S: # Same as Callable[..., S] + ... + table_entry: TInt # Same as Tuple[int, Any] + + T = TypeVar('T', int, float, complex) + + Vec = Iterable[Tuple[T, T]] + + def inproduct(v: Vec[T]) -> T: + return sum(x*y for x, y in v) + + def dilate(v: Vec[T], scale: T) -> Vec[T]: + return ((x * scale, y * scale) for x, y in v) + + v1: Vec[int] = [] # Same as Iterable[Tuple[int, int]] + v2: Vec = [] # Same as Iterable[Tuple[Any, Any]] + v3: Vec[int, int] = [] # Error: Invalid alias, too many type arguments! + +Type aliases can be imported from modules just like other names. An +alias can also target another alias, although building complex chains +of aliases is not recommended -- this impedes code readability, thus +defeating the purpose of using aliases. Example: + +.. code-block:: python + + from typing import TypeVar, Generic, Optional + from example1 import AliasType + from example2 import Vec + + # AliasType and Vec are type aliases (Vec as defined above) + + def fun() -> AliasType: + ... + + T = TypeVar('T') + + class NewVec(Vec[T]): + ... + + for i, j in NewVec[int](): + ... + + OIntVec = Optional[Vec[int]] + +.. note:: + + A type alias does not define a new type. For generic type aliases + this means that variance of type variables used for alias definition does not + apply to aliases. A parameterized generic alias is treated simply as an original + type with the corresponding type variables substituted. diff --git a/docs/source/index.rst b/docs/source/index.rst index 35a2aff8827e..71a3a94669b2 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -32,11 +32,12 @@ Mypy is a static type checker for Python 3 and Python 2.7. type_inference_and_annotations kinds_of_types class_basics + protocols dynamic_typing - function_overloading casts duck_type_compatibility generics + more_types .. toctree:: :maxdepth: 2 @@ -45,17 +46,16 @@ Mypy is a static type checker for Python 3 and Python 2.7. command_line config_file mypy_daemon - supported_python_features - python36 - additional_features - + installed_packages .. toctree:: :maxdepth: 2 :caption: Miscellaneous - installed_packages common_issues + supported_python_features + python36 + additional_features faq revision_history diff --git a/docs/source/installed_packages.rst b/docs/source/installed_packages.rst index a4e8c9e6abb6..c693c640ad34 100644 --- a/docs/source/installed_packages.rst +++ b/docs/source/installed_packages.rst @@ -1,6 +1,6 @@ .. _installed-packages: -Using Installed Packages +Using installed packages ======================== `PEP 561 `_ specifies how to mark diff --git a/docs/source/kinds_of_types.rst b/docs/source/kinds_of_types.rst index 9f7eb5c53ccd..596e0711a374 100644 --- a/docs/source/kinds_of_types.rst +++ b/docs/source/kinds_of_types.rst @@ -1,17 +1,23 @@ Kinds of types ============== -User-defined types -****************** +We've mostly restricted ourselves to built-in types until now. This +section introduces several additional kinds of types. You are likely +to need at least some of them to type check any non-trivial programs. -Each class is also a type. Any instance of a subclass is also -compatible with all superclasses. All values are compatible with the -``object`` type (and also the ``Any`` type). +Class types +*********** + +Every class is also a valid type. Any instance of a subclass is also +compatible with all superclasses -- it follows that every value is compatible +with the ``object`` type (and incidentally also the ``Any`` type, discussed +below). Mypy analyzes the bodies of classes to determine which methods and +attributes are available in instances. This example uses subclassing: .. code-block:: python class A: - def f(self) -> int: # Type of self inferred (A) + def f(self) -> int: # Type of self inferred (A) return 2 class B(A): @@ -20,34 +26,37 @@ compatible with all superclasses. All values are compatible with the def g(self) -> int: return 4 - a = B() # type: A # OK (explicit type for a; override type inference) - print(a.f()) # 3 - a.g() # Type check error: A has no method g + def foo(a: A) -> None: + print(a.f()) # 3 + a.g() # Error: "A" has no attribute "g" + + foo(B()) # OK (B is a subclass of A) The Any type ************ A value with the ``Any`` type is dynamically typed. Mypy doesn't know anything about the possible runtime types of such value. Any -operations are permitted on the value, and the operations are checked -at runtime, similar to normal Python code without type annotations. +operations are permitted on the value, and the operations are only checked +at runtime. You can use ``Any`` as an "escape hatch" when you can't use +a more precise type for some reason. -``Any`` is compatible with every other type, and vice versa. No -implicit type check is inserted when assigning a value of type ``Any`` -to a variable with a more precise type: +``Any`` is compatible with every other type, and vice versa. You can freely +assign a value of type ``Any`` to a variable with a more precise type: .. code-block:: python - a = None # type: Any - s = '' # type: str - a = 2 # OK - s = a # OK + a: Any = None + s: str = '' + a = 2 # OK (assign "int" to "Any") + s = a # OK (assign "Any" to "str") -Declared (and inferred) types are *erased* at runtime. They are +Declared (and inferred) types are ignored (or *erased*) at runtime. They are basically treated as comments, and thus the above code does not generate a runtime error, even though ``s`` gets an ``int`` value when -the program is run. Note that the declared type of ``s`` is actually -``str``! +the program is run, while the declared type of ``s`` is actually +``str``! You need to be careful with ``Any`` types, since they let you +lie to mypy, and this could easily hide bugs. If you do not define a function return value or argument types, these default to ``Any``: @@ -130,9 +139,9 @@ purpose. Example: .. note:: - ``Tuple[...]`` is not valid as a base class outside stub files. This is a - limitation of the ``typing`` module. One way to work around - this is to use a named tuple as a base class (see section :ref:`named-tuples`). + ``Tuple[...]`` is valid as a base class in Python 3.6 and later, and + always in stub files. In earlier Python versions you can sometimes work around this + limitation by using a named tuple as a base class (see section :ref:`named-tuples`). .. _callable-types: @@ -188,145 +197,21 @@ using bidirectional type inference: If you want to give the argument or return value types explicitly, use an ordinary, perhaps nested function definition. -.. _extended_callable: - -Extended Callable types -*********************** - -As an experimental mypy extension, you can specify ``Callable`` types -that support keyword arguments, optional arguments, and more. Where -you specify the arguments of a Callable, you can choose to supply just -the type of a nameless positional argument, or an "argument specifier" -representing a more complicated form of argument. This allows one to -more closely emulate the full range of possibilities given by the -``def`` statement in Python. - -As an example, here's a complicated function definition and the -corresponding ``Callable``: - -.. code-block:: python - - from typing import Callable - from mypy_extensions import (Arg, DefaultArg, NamedArg, - DefaultNamedArg, VarArg, KwArg) - - def func(__a: int, # This convention is for nameless arguments - b: int, - c: int = 0, - *args: int, - d: int, - e: int = 0, - **kwargs: int) -> int: - ... - - F = Callable[[int, # Or Arg(int) - Arg(int, 'b'), - DefaultArg(int, 'c'), - VarArg(int), - NamedArg(int, 'd'), - DefaultNamedArg(int, 'e'), - KwArg(int)], - int] - - f: F = func - -Argument specifiers are special function calls that can specify the -following aspects of an argument: - -- its type (the only thing that the basic format supports) - -- its name (if it has one) - -- whether it may be omitted - -- whether it may or must be passed using a keyword - -- whether it is a ``*args`` argument (representing the remaining - positional arguments) - -- whether it is a ``**kwargs`` argument (representing the remaining - keyword arguments) - -The following functions are available in ``mypy_extensions`` for this -purpose: - -.. code-block:: python - - def Arg(type=Any, name=None): - # A normal, mandatory, positional argument. - # If the name is specified it may be passed as a keyword. - - def DefaultArg(type=Any, name=None): - # An optional positional argument (i.e. with a default value). - # If the name is specified it may be passed as a keyword. - - def NamedArg(type=Any, name=None): - # A mandatory keyword-only argument. - - def DefaultNamedArg(type=Any, name=None): - # An optional keyword-only argument (i.e. with a default value). - - def VarArg(type=Any): - # A *args-style variadic positional argument. - # A single VarArg() specifier represents all remaining - # positional arguments. - - def KwArg(type=Any): - # A **kwargs-style variadic keyword argument. - # A single KwArg() specifier represents all remaining - # keyword arguments. - -In all cases, the ``type`` argument defaults to ``Any``, and if the -``name`` argument is omitted the argument has no name (the name is -required for ``NamedArg`` and ``DefaultNamedArg``). A basic -``Callable`` such as - -.. code-block:: python - - MyFunc = Callable[[int, str, int], float] - -is equivalent to the following: - -.. code-block:: python - - MyFunc = Callable[[Arg(int), Arg(str), Arg(int)], float] - -A ``Callable`` with unspecified argument types, such as - -.. code-block:: python - - MyOtherFunc = Callable[..., int] - -is (roughly) equivalent to - -.. code-block:: python - - MyOtherFunc = Callable[[VarArg(), KwArg()], int] - -.. note:: - - This feature is experimental. Details of the implementation may - change and there may be unknown limitations. **IMPORTANT:** - Each of the functions above currently just returns its ``type`` - argument, so the information contained in the argument specifiers - is not available at runtime. This limitation is necessary for - backwards compatibility with the existing ``typing.py`` module as - present in the Python 3.5+ standard library and distributed via - PyPI. - .. _union-types: Union types *********** Python functions often accept values of two or more different -types. You can use overloading to model this in statically typed code, -but union types can make code like this easier to write. +types. You can use :ref:`overloading ` to +represent this, but union types are often more convenient. Use the ``Union[T1, ..., Tn]`` type constructor to construct a union -type. For example, the type ``Union[int, str]`` is compatible with -both integers and strings. You can use an ``isinstance()`` check to -narrow down the type to a specific type: +type. For example, if an argument has type ``Union[int, str]``, both +integers and strings are valid argument values. + +You can use an ``isinstance()`` check to narrow down a union type to a +more specific type: .. code-block:: python @@ -345,6 +230,15 @@ narrow down the type to a specific type: f('x') # OK f(1.1) # Error +.. note:: + + Operations are valid for union types only if they are valid for *every* + union item. This is why it's often necessary to use an ``isinstance()`` + check to first narrow down a union type to a non-union type. This also + means that it's recommended to avoid union types as function return types, + since the caller may have to use ``isinstance()`` before doing anything + interesting with the value. + .. _strict_optional: Optional types and the None type @@ -579,48 +473,6 @@ valid for any type, but it's much more useful for a programmer who is reading the code. This also makes it easier to migrate to strict ``None`` checking in the future. -.. _noreturn: - -The NoReturn type -***************** - -Mypy provides support for functions that never return. For -example, a function that unconditionally raises an exception: - -.. code-block:: python - - from mypy_extensions import NoReturn - - def stop() -> NoReturn: - raise Exception('no way') - -Mypy will ensure that functions annotated as returning ``NoReturn`` -truly never return, either implicitly or explicitly. Mypy will also -recognize that the code after calls to such functions is unreachable -and will behave accordingly: - -.. code-block:: python - - def f(x: int) -> int: - if x == 0: - return x - stop() - return 'whatever works' # No error in an unreachable block - -Install ``mypy_extensions`` using pip to use ``NoReturn`` in your code. -Python 3 command line: - -.. code-block:: text - - python3 -m pip install --upgrade mypy-extensions - -This works for Python 2: - -.. code-block:: text - - pip install --upgrade mypy-extensions - - Class name forward references ***************************** @@ -691,187 +543,10 @@ assigning the type to a variable: def f() -> AliasType: ... -Type aliases can be generic, in this case they could be used in two variants: -Subscripted aliases are equivalent to original types with substituted type variables, -number of type arguments must match the number of free type variables -in generic type alias. Unsubscripted aliases are treated as original types with free -variables replaced with ``Any``. Examples (following `PEP 484 -`_): - -.. code-block:: python - - from typing import TypeVar, Iterable, Tuple, Union, Callable - S = TypeVar('S') - TInt = Tuple[int, S] - UInt = Union[S, int] - CBack = Callable[..., S] - - def response(query: str) -> UInt[str]: # Same as Union[str, int] - ... - def activate(cb: CBack[S]) -> S: # Same as Callable[..., S] - ... - table_entry: TInt # Same as Tuple[int, Any] - - T = TypeVar('T', int, float, complex) - Vec = Iterable[Tuple[T, T]] - - def inproduct(v: Vec[T]) -> T: - return sum(x*y for x, y in v) - - def dilate(v: Vec[T], scale: T) -> Vec[T]: - return ((x * scale, y * scale) for x, y in v) - - v1: Vec[int] = [] # Same as Iterable[Tuple[int, int]] - v2: Vec = [] # Same as Iterable[Tuple[Any, Any]] - v3: Vec[int, int] = [] # Error: Invalid alias, too many type arguments! - -Type aliases can be imported from modules like any names. Aliases can target another -aliases (although building complex chains of aliases is not recommended, this -impedes code readability, thus defeating the purpose of using aliases). -Following previous examples: - -.. code-block:: python - - from typing import TypeVar, Generic, Optional - from first_example import AliasType - from second_example import Vec - - def fun() -> AliasType: - ... - - T = TypeVar('T') - class NewVec(Generic[T], Vec[T]): - ... - for i, j in NewVec[int](): - ... - - OIntVec = Optional[Vec[int]] - .. note:: A type alias does not create a new type. It's just a shorthand notation for - another type -- it's equivalent to the target type. For generic type aliases - this means that variance of type variables used for alias definition does not - apply to aliases. A parameterized generic alias is treated simply as an original - type with the corresponding type variables substituted. - -.. _newtypes: - -NewTypes -******** - -(Freely after `PEP 484 -`_.) - -There are also situations where a programmer might want to avoid logical errors by -creating simple classes. For example: - -.. code-block:: python - - class UserId(int): - pass - - get_by_user_id(user_id: UserId): - ... - -However, this approach introduces some runtime overhead. To avoid this, the typing -module provides a helper function ``NewType`` that creates simple unique types with -almost zero runtime overhead. Mypy will treat the statement -``Derived = NewType('Derived', Base)`` as being roughly equivalent to the following -definition: - -.. code-block:: python - - class Derived(Base): - def __init__(self, _x: Base) -> None: - ... - -However, at runtime, ``NewType('Derived', Base)`` will return a dummy function that -simply returns its argument: - -.. code-block:: python - - def Derived(_x): - return _x - -Mypy will require explicit casts from ``int`` where ``UserId`` is expected, while -implicitly casting from ``UserId`` where ``int`` is expected. Examples: - -.. code-block:: python - - from typing import NewType - - UserId = NewType('UserId', int) - - def name_by_id(user_id: UserId) -> str: - ... - - UserId('user') # Fails type check - - name_by_id(42) # Fails type check - name_by_id(UserId(42)) # OK - - num = UserId(5) + 1 # type: int - -``NewType`` accepts exactly two arguments. The first argument must be a string literal -containing the name of the new type and must equal the name of the variable to which the new -type is assigned. The second argument must be a properly subclassable class, i.e., -not a type construct like ``Union``, etc. - -The function returned by ``NewType`` accepts only one argument; this is equivalent to -supporting only one constructor accepting an instance of the base class (see above). -Example: - -.. code-block:: python - - from typing import NewType - - class PacketId: - def __init__(self, major: int, minor: int) -> None: - self._major = major - self._minor = minor - - TcpPacketId = NewType('TcpPacketId', PacketId) - - packet = PacketId(100, 100) - tcp_packet = TcpPacketId(packet) # OK - - tcp_packet = TcpPacketId(127, 0) # Fails in type checker and at runtime - -Both ``isinstance`` and ``issubclass``, as well as subclassing will fail for -``NewType('Derived', Base)`` since function objects don't support these operations. - -.. note:: - - Note that unlike type aliases, ``NewType`` will create an entirely new and - unique type when used. The intended purpose of ``NewType`` is to help you - detect cases where you accidentally mixed together the old base type and the - new derived type. - - For example, the following will successfully typecheck when using type - aliases: - - .. code-block:: python - - UserId = int - - def name_by_id(user_id: UserId) -> str: - ... - - name_by_id(3) # ints and UserId are synonymous - - But a similar example using ``NewType`` will not typecheck: - - .. code-block:: python - - from typing import NewType - - UserId = NewType('UserId', int) - - def name_by_id(user_id: UserId) -> str: - ... - - name_by_id(3) # int is not the same as UserId + another type -- it's equivalent to the target type. .. _named-tuples: @@ -901,8 +576,7 @@ item types: ('y', int)]) p = Point(x=1, y='x') # Argument has incompatible type "str"; expected "int" -Python 3.6 will have an alternative, class-based syntax for named tuples with types. -Mypy supports it already: +Python 3.6 introduced an alternative, class-based syntax for named tuples with types: .. code-block:: python @@ -1088,376 +762,3 @@ This is slightly different from using ``Iterable[int]`` or ``Iterator[int]``, since generators have ``close()``, ``send()``, and ``throw()`` methods that generic iterables don't. If you will call these methods on the returned generator, use the ``Generator`` type instead of ``Iterable`` or ``Iterator``. - -.. _async-and-await: - -Typing async/await -****************** - -Mypy supports the ability to type coroutines that use the ``async/await`` -syntax introduced in Python 3.5. For more information regarding coroutines and -this new syntax, see `PEP 492 `_. - -Functions defined using ``async def`` are typed just like normal functions. -The return type annotation should be the same as the type of the value you -expect to get back when ``await``-ing the coroutine. - -.. code-block:: python - - import asyncio - - async def format_string(tag: str, count: int) -> str: - return 'T-minus {} ({})'.format(count, tag) - - async def countdown_1(tag: str, count: int) -> str: - while count > 0: - my_str = await format_string(tag, count) # has type 'str' - print(my_str) - await asyncio.sleep(0.1) - count -= 1 - return "Blastoff!" - - loop = asyncio.get_event_loop() - loop.run_until_complete(countdown_1("Millennium Falcon", 5)) - loop.close() - -The result of calling an ``async def`` function *without awaiting* will be a -value of type ``Awaitable[T]``: - -.. code-block:: python - - my_coroutine = countdown_1("Millennium Falcon", 5) - reveal_type(my_coroutine) # has type 'Awaitable[str]' - -.. note:: - - :ref:`reveal_type() ` displays the inferred static type of - an expression. - -If you want to use coroutines in older versions of Python that do not support -the ``async def`` syntax, you can instead use the ``@asyncio.coroutine`` -decorator to convert a generator into a coroutine. - -Note that we set the ``YieldType`` of the generator to be ``Any`` in the -following example. This is because the exact yield type is an implementation -detail of the coroutine runner (e.g. the ``asyncio`` event loop) and your -coroutine shouldn't have to know or care about what precisely that type is. - -.. code-block:: python - - from typing import Any, Generator - import asyncio - - @asyncio.coroutine - def countdown_2(tag: str, count: int) -> Generator[Any, None, str]: - while count > 0: - print('T-minus {} ({})'.format(count, tag)) - yield from asyncio.sleep(0.1) - count -= 1 - return "Blastoff!" - - loop = asyncio.get_event_loop() - loop.run_until_complete(countdown_2("USS Enterprise", 5)) - loop.close() - -As before, the result of calling a generator decorated with ``@asyncio.coroutine`` -will be a value of type ``Awaitable[T]``. - -.. note:: - - At runtime, you are allowed to add the ``@asyncio.coroutine`` decorator to - both functions and generators. This is useful when you want to mark a - work-in-progress function as a coroutine, but have not yet added ``yield`` or - ``yield from`` statements: - - .. code-block:: python - - import asyncio - - @asyncio.coroutine - def serialize(obj: object) -> str: - # todo: add yield/yield from to turn this into a generator - return "placeholder" - - However, mypy currently does not support converting functions into - coroutines. Support for this feature will be added in a future version, but - for now, you can manually force the function to be a generator by doing - something like this: - - .. code-block:: python - - from typing import Generator - import asyncio - - @asyncio.coroutine - def serialize(obj: object) -> Generator[None, None, str]: - # todo: add yield/yield from to turn this into a generator - if False: - yield - return "placeholder" - -You may also choose to create a subclass of ``Awaitable`` instead: - -.. code-block:: python - - from typing import Any, Awaitable, Generator - import asyncio - - class MyAwaitable(Awaitable[str]): - def __init__(self, tag: str, count: int) -> None: - self.tag = tag - self.count = count - - def __await__(self) -> Generator[Any, None, str]: - for i in range(n, 0, -1): - print('T-minus {} ({})'.format(i, tag)) - yield from asyncio.sleep(0.1) - return "Blastoff!" - - def countdown_3(tag: str, count: int) -> Awaitable[str]: - return MyAwaitable(tag, count) - - loop = asyncio.get_event_loop() - loop.run_until_complete(countdown_3("Heart of Gold", 5)) - loop.close() - -To create an iterable coroutine, subclass ``AsyncIterator``: - -.. code-block:: python - - from typing import Optional, AsyncIterator - import asyncio - - class arange(AsyncIterator[int]): - def __init__(self, start: int, stop: int, step: int) -> None: - self.start = start - self.stop = stop - self.step = step - self.count = start - step - - def __aiter__(self) -> AsyncIterator[int]: - return self - - async def __anext__(self) -> int: - self.count += self.step - if self.count == self.stop: - raise StopAsyncIteration - else: - return self.count - - async def countdown_4(tag: str, n: int) -> str: - async for i in arange(n, 0, -1): - print('T-minus {} ({})'.format(i, tag)) - await asyncio.sleep(0.1) - return "Blastoff!" - - loop = asyncio.get_event_loop() - loop.run_until_complete(countdown_4("Serenity", 5)) - loop.close() - -For a more concrete example, the mypy repo has a toy webcrawler that -demonstrates how to work with coroutines. One version -`uses async/await `_ -and one -`uses yield from `_. - -.. _typeddict: - -TypedDict -********* - -.. note:: - - TypedDict is an officially supported feature, but it is still experimental. - - -Python programs often use dictionaries with string keys to represent objects. -Here is a typical example: - -.. code-block:: python - - movie = {'name': 'Blade Runner', 'year': 1982} - -Only a fixed set of string keys is expected (``'name'`` and -``'year'`` above), and each key has an independent value type (``str`` -for ``'name'`` and ``int`` for ``'year'`` above). We've previously -seen the ``Dict[K, V]`` type, which lets you declare uniform -dictionary types, where every value has the same type, and arbitrary keys -are supported. This is clearly not a good fit for -``movie`` above. Instead, you can use a ``TypedDict`` to give a precise -type for objects like ``movie``, where the type of each -dictionary value depends on the key: - -.. code-block:: python - - from mypy_extensions import TypedDict - - Movie = TypedDict('Movie', {'name': str, 'year': int}) - - movie = {'name': 'Blade Runner', 'year': 1982} # type: Movie - -``Movie`` is a TypedDict type with two items: ``'name'`` (with type ``str``) -and ``'year'`` (with type ``int``). Note that we used an explicit type -annotation for the ``movie`` variable. This type annotation is -important -- without it, mypy will try to infer a regular, uniform -``Dict`` type for ``movie``, which is not what we want here. - -.. note:: - - If you pass a TypedDict object as an argument to a function, no - type annotation is usually necessary since mypy can infer the - desired type based on the declared argument type. Also, if an - assignment target has been previously defined, and it has a - TypedDict type, mypy will treat the assigned value as a TypedDict, - not ``Dict``. - -Now mypy will recognize these as valid: - -.. code-block:: python - - name = movie['name'] # Okay; type of name is str - year = movie['year'] # Okay; type of year is int - -Mypy will detect an invalid key as an error: - -.. code-block:: python - - director = movie['director'] # Error: 'director' is not a valid key - -Mypy will also reject a runtime-computed expression as a key, as -it can't verify that it's a valid key. You can only use string -literals as TypedDict keys. - -The ``TypedDict`` type object can also act as a constructor. It -returns a normal ``dict`` object at runtime -- a ``TypedDict`` does -not define a new runtime type: - -.. code-block:: python - - toy_story = Movie(name='Toy Story', year=1995) - -This is equivalent to just constructing a dictionary directly using -``{ ... }`` or ``dict(key=value, ...)``. The constructor form is -sometimes convenient, since it can be used without a type annotation, -and it also makes the type of the object explicit. - -Like all types, TypedDicts can be used as components to build -arbitrarily complex types. For example, you can define nested -TypedDicts and containers with TypedDict items. -Unlike most other types, mypy uses structural compatibility checking -(or structural subtyping) with TypedDicts. A TypedDict object with -extra items is compatible with a narrower TypedDict, assuming item -types are compatible (*totality* also affects -subtyping, as discussed below). - -.. note:: - - You need to install ``mypy_extensions`` using pip to use ``TypedDict``: - - .. code-block:: text - - python3 -m pip install --upgrade mypy-extensions - - Or, if you are using Python 2: - - .. code-block:: text - - pip install --upgrade mypy-extensions - -Totality --------- - -By default mypy ensures that a TypedDict object has all the specified -keys. This will be flagged as an error: - -.. code-block:: python - - # Error: 'year' missing - toy_story = {'name': 'Toy Story'} # type: Movie - -Sometimes you want to allow keys to be left out when creating a -TypedDict object. You can provide the ``total=False`` argument to -``TypedDict(...)`` to achieve this: - -.. code-block:: python - - GuiOptions = TypedDict( - 'GuiOptions', {'language': str, 'color': str}, total=False) - options = {} # type: GuiOptions # Okay - options['language'] = 'en' - -You may need to use ``get()`` to access items of a partial (non-total) -TypedDict, since indexing using ``[]`` could fail at runtime. -However, mypy still lets use ``[]`` with a partial TypedDict -- you -just need to be careful with it, as it could result in a ``KeyError``. -Requiring ``get()`` everywhere would be too cumbersome. (Note that you -are free to use ``get()`` with total TypedDicts as well.) - -Keys that aren't required are shown with a ``?`` in error messages: - -.. code-block:: python - - # Revealed type is 'TypedDict('GuiOptions', {'language'?: builtins.str, - # 'color'?: builtins.str})' - reveal_type(options) - -Totality also affects structural compatibility. You can't use a partial -TypedDict when a total one is expected. Also, a total typed dict is not -valid when a partial one is expected. - -Class-based syntax ------------------- - -Python 3.6 supports an alternative, class-based syntax to define a -TypedDict. This means that your code must be checked as if it were -Python 3.6 (using the ``--python-version`` flag on the command line, -for example). Simply running mypy on Python 3.6 is insufficient. - -.. code-block:: python - - from mypy_extensions import TypedDict - - class Movie(TypedDict): - name: str - year: int - -The above definition is equivalent to the original ``Movie`` -definition. It doesn't actually define a real class. This syntax also -supports a form of inheritance -- subclasses can define additional -items. However, this is primarily a notational shortcut. Since mypy -uses structural compatibility with TypedDicts, inheritance is not -required for compatibility. Here is an example of inheritance: - -.. code-block:: python - - class Movie(TypedDict): - name: str - year: int - - class BookBasedMovie(Movie): - based_on: str - -Now ``BookBasedMovie`` has keys ``name``, ``year`` and ``based_on``. - -Mixing required and non-required items --------------------------------------- - -In addition to allowing reuse across TypedDict types, inheritance also allows -you to mix required and non-required (using ``total=False``) items -in a single TypedDict. Example: - -.. code-block:: python - - class MovieBase(TypedDict): - name: str - year: int - - class Movie(MovieBase, total=False): - based_on: str - -Now ``Movie`` has required keys ``name`` and ``year``, while ``based_on`` -can be left out when constructing an object. A TypedDict with a mix of required -and non-required keys, such as ``Movie`` above, will only be compatible with -another TypedDict if all required keys in the other TypedDict are required keys in the -first TypedDict, and all non-required keys of the other TypedDict are also non-required keys -in the first TypedDict. diff --git a/docs/source/more_types.rst b/docs/source/more_types.rst new file mode 100644 index 000000000000..61e4f4dc7b76 --- /dev/null +++ b/docs/source/more_types.rst @@ -0,0 +1,551 @@ +More types +========== + +This section introduces a few additional kinds of types, including ``NoReturn``, +``NewType``, ``TypedDict``, and types for async code. All of these are only +situationally useful, so feel free to skip this section and come back when you +have a need for some of them. + +Here's a quick summary of what's covered here: + +* ``NoReturn`` lets you tell mypy that a function never returns normally. + +* ``NewType`` lets you define a variant of a type that is treated as a + separate type by mypy but is identical to the original type at runtime. + For example, you can have ``UserId`` as a variant of ``int`` that is + just an ``int`` at runtime. + +* ``TypedDict`` lets you give precise types for dictionaries that represent + objects with a fixed schema, such as ``{'id': 1, 'items': ['x']}``. + +* Async types let you type check programs using ``async`` and ``await``. + +.. _noreturn: + +The NoReturn type +***************** + +Mypy provides support for functions that never return. For +example, a function that unconditionally raises an exception: + +.. code-block:: python + + from typing import NoReturn + + def stop() -> NoReturn: + raise Exception('no way') + +Mypy will ensure that functions annotated as returning ``NoReturn`` +truly never return, either implicitly or explicitly. Mypy will also +recognize that the code after calls to such functions is unreachable +and will behave accordingly: + +.. code-block:: python + + def f(x: int) -> int: + if x == 0: + return x + stop() + return 'whatever works' # No error in an unreachable block + +In earlier Python versions you need to install ``typing_extensions`` using +pip to use ``NoReturn`` in your code. Python 3 command line: + +.. code-block:: text + + python3 -m pip install --upgrade typing-extensions + +This works for Python 2: + +.. code-block:: text + + pip install --upgrade typing-extensions + +.. _newtypes: + +NewTypes +******** + +There are situations where you may want to avoid programming errors by +creating simple derived classes that are only used to distinguish +certain values from base class instances. Example: + +.. code-block:: python + + class UserId(int): + pass + + get_by_user_id(user_id: UserId): + ... + +However, this approach introduces some runtime overhead. To avoid this, the typing +module provides a helper function ``NewType`` that creates simple unique types with +almost zero runtime overhead. Mypy will treat the statement +``Derived = NewType('Derived', Base)`` as being roughly equivalent to the following +definition: + +.. code-block:: python + + class Derived(Base): + def __init__(self, _x: Base) -> None: + ... + +However, at runtime, ``NewType('Derived', Base)`` will return a dummy function that +simply returns its argument: + +.. code-block:: python + + def Derived(_x): + return _x + +Mypy will require explicit casts from ``int`` where ``UserId`` is expected, while +implicitly casting from ``UserId`` where ``int`` is expected. Examples: + +.. code-block:: python + + from typing import NewType + + UserId = NewType('UserId', int) + + def name_by_id(user_id: UserId) -> str: + ... + + UserId('user') # Fails type check + + name_by_id(42) # Fails type check + name_by_id(UserId(42)) # OK + + num = UserId(5) + 1 # type: int + +``NewType`` accepts exactly two arguments. The first argument must be a string literal +containing the name of the new type and must equal the name of the variable to which the new +type is assigned. The second argument must be a properly subclassable class, i.e., +not a type construct like ``Union``, etc. + +The function returned by ``NewType`` accepts only one argument; this is equivalent to +supporting only one constructor accepting an instance of the base class (see above). +Example: + +.. code-block:: python + + from typing import NewType + + class PacketId: + def __init__(self, major: int, minor: int) -> None: + self._major = major + self._minor = minor + + TcpPacketId = NewType('TcpPacketId', PacketId) + + packet = PacketId(100, 100) + tcp_packet = TcpPacketId(packet) # OK + + tcp_packet = TcpPacketId(127, 0) # Fails in type checker and at runtime + +You cannot use ``isinstance()`` or ``issubclass()`` on the object returned by +``NewType()``, because function objects don't support these operations. You cannot +create subclasses of these objects either. + +.. note:: + + Unlike type aliases, ``NewType`` will create an entirely new and + unique type when used. The intended purpose of ``NewType`` is to help you + detect cases where you accidentally mixed together the old base type and the + new derived type. + + For example, the following will successfully typecheck when using type + aliases: + + .. code-block:: python + + UserId = int + + def name_by_id(user_id: UserId) -> str: + ... + + name_by_id(3) # ints and UserId are synonymous + + But a similar example using ``NewType`` will not typecheck: + + .. code-block:: python + + from typing import NewType + + UserId = NewType('UserId', int) + + def name_by_id(user_id: UserId) -> str: + ... + + name_by_id(3) # int is not the same as UserId + +.. _async-and-await: + +Typing async/await +****************** + +Mypy supports the ability to type coroutines that use the ``async/await`` +syntax introduced in Python 3.5. For more information regarding coroutines and +this new syntax, see `PEP 492 `_. + +Functions defined using ``async def`` are typed just like normal functions. +The return type annotation should be the same as the type of the value you +expect to get back when ``await``-ing the coroutine. + +.. code-block:: python + + import asyncio + + async def format_string(tag: str, count: int) -> str: + return 'T-minus {} ({})'.format(count, tag) + + async def countdown_1(tag: str, count: int) -> str: + while count > 0: + my_str = await format_string(tag, count) # has type 'str' + print(my_str) + await asyncio.sleep(0.1) + count -= 1 + return "Blastoff!" + + loop = asyncio.get_event_loop() + loop.run_until_complete(countdown_1("Millennium Falcon", 5)) + loop.close() + +The result of calling an ``async def`` function *without awaiting* will be a +value of type ``typing.Coroutine[Any, Any, T]``, which is a subtype of +``Awaitable[T]``: + +.. code-block:: python + + my_coroutine = countdown_1("Millennium Falcon", 5) + reveal_type(my_coroutine) # has type 'Coroutine[Any, Any, str]' + +.. note:: + + :ref:`reveal_type() ` displays the inferred static type of + an expression. + +If you want to use coroutines in Python 3.4, which does not support +the ``async def`` syntax, you can instead use the ``@asyncio.coroutine`` +decorator to convert a generator into a coroutine. + +Note that we set the ``YieldType`` of the generator to be ``Any`` in the +following example. This is because the exact yield type is an implementation +detail of the coroutine runner (e.g. the ``asyncio`` event loop) and your +coroutine shouldn't have to know or care about what precisely that type is. + +.. code-block:: python + + from typing import Any, Generator + import asyncio + + @asyncio.coroutine + def countdown_2(tag: str, count: int) -> Generator[Any, None, str]: + while count > 0: + print('T-minus {} ({})'.format(count, tag)) + yield from asyncio.sleep(0.1) + count -= 1 + return "Blastoff!" + + loop = asyncio.get_event_loop() + loop.run_until_complete(countdown_2("USS Enterprise", 5)) + loop.close() + +As before, the result of calling a generator decorated with ``@asyncio.coroutine`` +will be a value of type ``Awaitable[T]``. + +.. note:: + + At runtime, you are allowed to add the ``@asyncio.coroutine`` decorator to + both functions and generators. This is useful when you want to mark a + work-in-progress function as a coroutine, but have not yet added ``yield`` or + ``yield from`` statements: + + .. code-block:: python + + import asyncio + + @asyncio.coroutine + def serialize(obj: object) -> str: + # todo: add yield/yield from to turn this into a generator + return "placeholder" + + However, mypy currently does not support converting functions into + coroutines. Support for this feature will be added in a future version, but + for now, you can manually force the function to be a generator by doing + something like this: + + .. code-block:: python + + from typing import Generator + import asyncio + + @asyncio.coroutine + def serialize(obj: object) -> Generator[None, None, str]: + # todo: add yield/yield from to turn this into a generator + if False: + yield + return "placeholder" + +You may also choose to create a subclass of ``Awaitable`` instead: + +.. code-block:: python + + from typing import Any, Awaitable, Generator + import asyncio + + class MyAwaitable(Awaitable[str]): + def __init__(self, tag: str, count: int) -> None: + self.tag = tag + self.count = count + + def __await__(self) -> Generator[Any, None, str]: + for i in range(n, 0, -1): + print('T-minus {} ({})'.format(i, tag)) + yield from asyncio.sleep(0.1) + return "Blastoff!" + + def countdown_3(tag: str, count: int) -> Awaitable[str]: + return MyAwaitable(tag, count) + + loop = asyncio.get_event_loop() + loop.run_until_complete(countdown_3("Heart of Gold", 5)) + loop.close() + +To create an iterable coroutine, subclass ``AsyncIterator``: + +.. code-block:: python + + from typing import Optional, AsyncIterator + import asyncio + + class arange(AsyncIterator[int]): + def __init__(self, start: int, stop: int, step: int) -> None: + self.start = start + self.stop = stop + self.step = step + self.count = start - step + + def __aiter__(self) -> AsyncIterator[int]: + return self + + async def __anext__(self) -> int: + self.count += self.step + if self.count == self.stop: + raise StopAsyncIteration + else: + return self.count + + async def countdown_4(tag: str, n: int) -> str: + async for i in arange(n, 0, -1): + print('T-minus {} ({})'.format(i, tag)) + await asyncio.sleep(0.1) + return "Blastoff!" + + loop = asyncio.get_event_loop() + loop.run_until_complete(countdown_4("Serenity", 5)) + loop.close() + +For a more concrete example, the mypy repo has a toy webcrawler that +demonstrates how to work with coroutines. One version +`uses async/await `_ +and one +`uses yield from `_. + +.. _typeddict: + +TypedDict +********* + +.. note:: + + TypedDict is an officially supported feature, but it is still experimental. + + +Python programs often use dictionaries with string keys to represent objects. +Here is a typical example: + +.. code-block:: python + + movie = {'name': 'Blade Runner', 'year': 1982} + +Only a fixed set of string keys is expected (``'name'`` and +``'year'`` above), and each key has an independent value type (``str`` +for ``'name'`` and ``int`` for ``'year'`` above). We've previously +seen the ``Dict[K, V]`` type, which lets you declare uniform +dictionary types, where every value has the same type, and arbitrary keys +are supported. This is clearly not a good fit for +``movie`` above. Instead, you can use a ``TypedDict`` to give a precise +type for objects like ``movie``, where the type of each +dictionary value depends on the key: + +.. code-block:: python + + from mypy_extensions import TypedDict + + Movie = TypedDict('Movie', {'name': str, 'year': int}) + + movie = {'name': 'Blade Runner', 'year': 1982} # type: Movie + +``Movie`` is a TypedDict type with two items: ``'name'`` (with type ``str``) +and ``'year'`` (with type ``int``). Note that we used an explicit type +annotation for the ``movie`` variable. This type annotation is +important -- without it, mypy will try to infer a regular, uniform +``Dict`` type for ``movie``, which is not what we want here. + +.. note:: + + If you pass a TypedDict object as an argument to a function, no + type annotation is usually necessary since mypy can infer the + desired type based on the declared argument type. Also, if an + assignment target has been previously defined, and it has a + TypedDict type, mypy will treat the assigned value as a TypedDict, + not ``Dict``. + +Now mypy will recognize these as valid: + +.. code-block:: python + + name = movie['name'] # Okay; type of name is str + year = movie['year'] # Okay; type of year is int + +Mypy will detect an invalid key as an error: + +.. code-block:: python + + director = movie['director'] # Error: 'director' is not a valid key + +Mypy will also reject a runtime-computed expression as a key, as +it can't verify that it's a valid key. You can only use string +literals as TypedDict keys. + +The ``TypedDict`` type object can also act as a constructor. It +returns a normal ``dict`` object at runtime -- a ``TypedDict`` does +not define a new runtime type: + +.. code-block:: python + + toy_story = Movie(name='Toy Story', year=1995) + +This is equivalent to just constructing a dictionary directly using +``{ ... }`` or ``dict(key=value, ...)``. The constructor form is +sometimes convenient, since it can be used without a type annotation, +and it also makes the type of the object explicit. + +Like all types, TypedDicts can be used as components to build +arbitrarily complex types. For example, you can define nested +TypedDicts and containers with TypedDict items. +Unlike most other types, mypy uses structural compatibility checking +(or structural subtyping) with TypedDicts. A TypedDict object with +extra items is compatible with a narrower TypedDict, assuming item +types are compatible (*totality* also affects +subtyping, as discussed below). + +.. note:: + + You need to install ``mypy_extensions`` using pip to use ``TypedDict``: + + .. code-block:: text + + python3 -m pip install --upgrade mypy-extensions + + Or, if you are using Python 2: + + .. code-block:: text + + pip install --upgrade mypy-extensions + +Totality +-------- + +By default mypy ensures that a TypedDict object has all the specified +keys. This will be flagged as an error: + +.. code-block:: python + + # Error: 'year' missing + toy_story = {'name': 'Toy Story'} # type: Movie + +Sometimes you want to allow keys to be left out when creating a +TypedDict object. You can provide the ``total=False`` argument to +``TypedDict(...)`` to achieve this: + +.. code-block:: python + + GuiOptions = TypedDict( + 'GuiOptions', {'language': str, 'color': str}, total=False) + options = {} # type: GuiOptions # Okay + options['language'] = 'en' + +You may need to use ``get()`` to access items of a partial (non-total) +TypedDict, since indexing using ``[]`` could fail at runtime. +However, mypy still lets use ``[]`` with a partial TypedDict -- you +just need to be careful with it, as it could result in a ``KeyError``. +Requiring ``get()`` everywhere would be too cumbersome. (Note that you +are free to use ``get()`` with total TypedDicts as well.) + +Keys that aren't required are shown with a ``?`` in error messages: + +.. code-block:: python + + # Revealed type is 'TypedDict('GuiOptions', {'language'?: builtins.str, + # 'color'?: builtins.str})' + reveal_type(options) + +Totality also affects structural compatibility. You can't use a partial +TypedDict when a total one is expected. Also, a total TypedDict is not +valid when a partial one is expected. + +Class-based syntax +------------------ + +An alternative, class-based syntax to define a TypedDict is supported +in Python 3.6 and later: + +.. code-block:: python + + from mypy_extensions import TypedDict + + class Movie(TypedDict): + name: str + year: int + +The above definition is equivalent to the original ``Movie`` +definition. It doesn't actually define a real class. This syntax also +supports a form of inheritance -- subclasses can define additional +items. However, this is primarily a notational shortcut. Since mypy +uses structural compatibility with TypedDicts, inheritance is not +required for compatibility. Here is an example of inheritance: + +.. code-block:: python + + class Movie(TypedDict): + name: str + year: int + + class BookBasedMovie(Movie): + based_on: str + +Now ``BookBasedMovie`` has keys ``name``, ``year`` and ``based_on``. + +Mixing required and non-required items +-------------------------------------- + +In addition to allowing reuse across TypedDict types, inheritance also allows +you to mix required and non-required (using ``total=False``) items +in a single TypedDict. Example: + +.. code-block:: python + + class MovieBase(TypedDict): + name: str + year: int + + class Movie(MovieBase, total=False): + based_on: str + +Now ``Movie`` has required keys ``name`` and ``year``, while ``based_on`` +can be left out when constructing an object. A TypedDict with a mix of required +and non-required keys, such as ``Movie`` above, will only be compatible with +another TypedDict if all required keys in the other TypedDict are required keys in the +first TypedDict, and all non-required keys of the other TypedDict are also non-required keys +in the first TypedDict. diff --git a/docs/source/protocols.rst b/docs/source/protocols.rst new file mode 100644 index 000000000000..113f73538e6c --- /dev/null +++ b/docs/source/protocols.rst @@ -0,0 +1,411 @@ +.. _protocol-types: + +Protocols and structural subtyping +================================== + +Mypy supports two ways of deciding whether two classes are compatible +as types: nominal subtyping and structural subtyping. *Nominal* +subtyping is strictly based on the class hierarchy. If class ``D`` +inherits class ``C``, it's also a subtype of ``C``, and instances of +``D`` can be used when ``C`` instances are expected. This form of +subtyping is used by default in mypy, since it's easy to understand +and produces clear and concise error messages, and since it matches +how the native ``isinstance()`` check works -- based on class +hierarchy. *Structural* subtyping can also be useful. Class ``D`` is +a structural subtype of class ``C`` if the former has all attributes +and methods of the latter, and with compatible types. + +Structural subtyping can be seen as a static equivalent of duck +typing, which is well known to Python programmers. Mypy provides +support for structural subtyping via protocol classes described +below. See `PEP 544 `_ for +the detailed specification of protocols and structural subtyping in +Python. + +.. _predefined_protocols: + +Predefined protocols +******************** + +The ``typing`` module defines various protocol classes that correspond +to common Python protocols, such as ``Iterable[T]``. If a class +defines a suitable ``__iter__`` method, mypy understands that it +implements the iterable protocol and is compatible with ``Iterable[T]``. +For example, ``IntList`` below is iterable, over ``int`` values: + +.. code-block:: python + + from typing import Iterator, Iterable, Optional + + class IntList: + def __init__(self, value: int, next: Optional[IntList]) -> None: + self.value = value + self.next = next + + def __iter__(self) -> Iterator[int]: + current = self + while current: + yield current.value + current = current.next + + def print_numbered(items: Iterable[int]) -> None: + for n, x in enumerate(items): + print(n + 1, x) + + x = IntList(3, IntList(5, None)) + print_numbered(x) # OK + print_numbered([4, 5]) # Also OK + +The subsections below introduce all built-in protocols defined in +``typing`` and the signatures of the corresponding methods you need to define +to implement each protocol (the signatures can be left out, as always, but mypy +won't type check unannotated methods). + +Iteration protocols +................... + +The iteration protocols are useful in many contexts. For example, they allow +iteration of objects in for loops. + +``Iterable[T]`` +--------------- + +The :ref:`example above ` has a simple implementation of an +``__iter__`` method. + +.. code-block:: python + + def __iter__(self) -> Iterator[T] + +``Iterator[T]`` +--------------- + +.. code-block:: python + + def __next__(self) -> T + def __iter__(self) -> Iterator[T] + +Collection protocols +.................... + +Many of these are implemented by built-in container types such as +``list`` and ``dict``, and these are also useful for user-defined +collection objects. + +``Sized`` +--------- + +This is a type for objects that support ``len(x)``. + +.. code-block:: python + + def __len__(self) -> int + +``Container[T]`` +---------------- + +This is a type for objects that support the ``in`` operator. + +.. code-block:: python + + def __contains__(self, x: object) -> bool + +``Collection[T]`` +----------------- + +.. code-block:: python + + def __len__(self) -> int + def __iter__(self) -> Iterator[T] + def __contains__(self, x: object) -> bool + +One-off protocols +................. + +These protocols are typically only useful with a single standard +library function or class. + +``Reversible[T]`` +----------------- + +This is a type for objects that support ``reversed(x)``. + +.. code-block:: python + + def __reversed__(self) -> Iterator[T] + +``SupportsAbs[T]`` +------------------ + +This is a type for objects that support ``abs(x)``. ``T`` is the type of +value returned by ``abs(x)``. + +.. code-block:: python + + def __abs__(self) -> T + +``SupportsBytes`` +----------------- + +This is a type for objects that support ``bytes(x)``. + +.. code-block:: python + + def __bytes__(self) -> bytes + +.. _supports-int-etc: + +``SupportsComplex`` +------------------- + +This is a type for objects that support ``complex(x)``. Note that no arithmetic operations +are supported. + +.. code-block:: python + + def __complex__(self) -> complex + +``SupportsFloat`` +----------------- + +This is a type for objects that support ``float(x)``. Note that no arithmetic operations +are supported. + +.. code-block:: python + + def __float__(self) -> float + +``SupportsInt`` +--------------- + +This is a type for objects that support ``int(x)``. Note that no arithmetic operations +are supported. + +.. code-block:: python + + def __int__(self) -> int + +``SupportsRound[T]`` +-------------------- + +This is a type for objects that support ``round(x)``. + +.. code-block:: python + + def __round__(self) -> T + +Async protocols +............... + +These protocols can be useful in async code. See :ref:`async-and-await` +for more information. + +``Awaitable[T]`` +---------------- + +.. code-block:: python + + def __await__(self) -> Generator[Any, None, T] + +``AsyncIterable[T]`` +-------------------- + +.. code-block:: python + + def __aiter__(self) -> AsyncIterator[T] + +``AsyncIterator[T]`` +-------------------- + +.. code-block:: python + + def __anext__(self) -> Awaitable[T] + def __aiter__(self) -> AsyncIterator[T] + +Context manager protocols +......................... + +There are two protocols for context managers -- one for regular context +managers and one for async ones. These allow defining objects that can +be used in ``with`` and ``async with`` statements. + +``ContextManager[T]`` +--------------------- + +.. code-block:: python + + def __enter__(self) -> T + def __exit__(self, + exc_type: Optional[Type[BaseException]], + exc_value: Optional[BaseException], + traceback: Optional[TracebackType]) -> Optional[bool] + +``AsyncContextManager[T]`` +-------------------------- + +.. code-block:: python + + def __aenter__(self) -> Awaitable[T] + def __aexit__(self, + exc_type: Optional[Type[BaseException]], + exc_value: Optional[BaseException], + traceback: Optional[TracebackType]) -> Awaitable[Optional[bool]] + +Simple user-defined protocols +***************************** + +You can define your own protocol class by inheriting the special +``typing_extensions.Protocol`` class: + +.. code-block:: python + + from typing import Iterable + from typing_extensions import Protocol + + class SupportsClose(Protocol): + def close(self) -> None: + ... # Empty method body (explicit '...') + + class Resource: # No SupportsClose base class! + # ... some methods ... + + def close(self) -> None: + self.resource.release() + + def close_all(items: Iterable[SupportsClose]) -> None: + for item in items: + item.close() + + close_all([Resource(), open('some/file')]) # Okay! + +``Resource`` is a subtype of the ``SupportsClose`` protocol since it defines +a compatible ``close`` method. Regular file objects returned by ``open()`` are +similarly compatible with the protocol, as they support ``close()``. + +.. note:: + + The ``Protocol`` base class is currently provided in the ``typing_extensions`` + package. Once structural subtyping is mature and + `PEP 544 `_ has been accepted, + ``Protocol`` will be included in the ``typing`` module. + +Defining subprotocols and subclassing protocols +*********************************************** + +You can also define subprotocols. Existing protocols can be extended +and merged using multiple inheritance. Example: + +.. code-block:: python + + # ... continuing from the previous example + + class SupportsRead(Protocol): + def read(self, amount: int) -> bytes: ... + + class TaggedReadableResource(SupportsClose, SupportsRead, Protocol): + label: str + + class AdvancedResource(Resource): + def __init__(self, label: str) -> None: + self.label = label + + def read(self, amount: int) -> bytes: + # some implementation + ... + + resource: TaggedReadableResource + resource = AdvancedResource('handle with care') # OK + +Note that inheriting from an existing protocol does not automatically +turn the subclass into a protocol -- it just creates a regular +(non-protocol) class or ABC that implements the given protocol (or +protocols). The ``typing_extensions.Protocol`` base class must always +be explicitly present if you are defining a protocol: + +.. code-block:: python + + class NotAProtocol(SupportsClose): # This is NOT a protocol + new_attr: int + + class Concrete: + new_attr: int = 0 + + def close(self) -> None: + ... + + # Error: nominal subtyping used by default + x: NotAProtocol = Concrete() # Error! + +You can also include default implementations of methods in +protocols. If you explicitly subclass these protocols you can inherit +these default implementations. Explicitly including a protocol as a +base class is also a way of documenting that your class implements a +particular protocol, and it forces mypy to verify that your class +implementation is actually compatible with the protocol. + +.. note:: + + You can use Python 3.6 variable annotations (`PEP 526 + `_) + to declare protocol attributes. On Python 2.7 and earlier Python 3 + versions you can use type comments and properties. + +Recursive protocols +******************* + +Protocols can be recursive (self-referential) and mutually +recursive. This is useful for declaring abstract recursive collections +such as trees and linked lists: + +.. code-block:: python + + from typing import TypeVar, Optional + from typing_extensions import Protocol + + class TreeLike(Protocol): + value: int + + @property + def left(self) -> Optional['TreeLike']: ... + + @property + def right(self) -> Optional['TreeLike']: ... + + class SimpleTree: + def __init__(self, value: int) -> None: + self.value = value + self.left: Optional['SimpleTree'] = None + self.right: Optional['SimpleTree'] = None + + root: TreeLike = SimpleTree(0) # OK + +Using ``isinstance()`` with protocols +************************************* + +You can use a protocol class with ``isinstance()`` if you decorate it +with the ``typing_extensions.runtime`` class decorator. The decorator +adds support for basic runtime structural checks: + +.. code-block:: python + + from typing_extensions import Protocol, runtime + + @runtime + class Portable(Protocol): + handles: int + + class Mug: + def __init__(self) -> None: + self.handles = 1 + + mug = Mug() + if isinstance(mug, Portable): + use(mug.handles) # Works statically and at runtime + +``isinstance()`` also works with the :ref:`predefined protocols ` +in ``typing`` such as ``Iterable``. + +.. note:: + ``isinstance()`` with protocols is not completely safe at runtime. + For example, signatures of methods are not checked. The runtime + implementation only checks that all protocol members are defined. diff --git a/docs/source/python36.rst b/docs/source/python36.rst index f1f73220cb7b..a5ffc3ddfa11 100644 --- a/docs/source/python36.rst +++ b/docs/source/python36.rst @@ -3,44 +3,36 @@ New features in Python 3.6 ========================== -Python 3.6 was `released -`_ in -December 2016. As of mypy 0.510 all language features new in Python -3.6 are supported. +Mypy has supported all language features new in Python 3.6 starting with mypy +0.510. This section introduces Python 3.6 features that interact with +type checking. Syntax for variable annotations (`PEP 526 `_) --------------------------------------------------------------------------------------- -Python 3.6 feature: variables (in global, class or local scope) can -now have type annotations using either of the two forms: +Python 3.6 introduced a new syntax for variable annotations (in +global, class and local scopes). There are two variants of the +syntax, with or without an initializer expression: .. code-block:: python from typing import Optional - foo: Optional[int] - bar: List[str] = [] - -Mypy fully supports this syntax, interpreting them as equivalent to - -.. code-block:: python - - foo = None # type: Optional[int] - bar = [] # type: List[str] + foo: Optional[int] # No initializer + bar: List[str] = [] # Initializer .. _class-var: -An additional feature defined in PEP 526 is also supported: you can -mark names intended to be used as class variables with ``ClassVar``. -In a pinch you can also use ClassVar in ``# type`` comments. -Example: +You can also mark names intended to be used as class variables with +``ClassVar``. In a pinch you can also use ClassVar in ``# type`` +comments. Example: .. code-block:: python from typing import ClassVar class C: - x: int # instance variable - y: ClassVar[int] # class variable + x: int # Instance variable + y: ClassVar[int] # Class variable z = None # type: ClassVar[int] def foo(self) -> None: @@ -50,37 +42,15 @@ Example: C.y = 0 # This is OK -Literal string formatting (`PEP 498 `_) ---------------------------------------------------------------------------------- - -Python 3.6 feature: string literals of the form -``f"text {expression} text"`` evaluate ``expression`` using the -current evaluation context (locals and globals). - -Mypy fully supports this syntax and type-checks the ``expression``. - -Underscores in numeric literals (`PEP 515 `_) ---------------------------------------------------------------------------------------- - -Python 3.6 feature: numeric literals can contain underscores, -e.g. ``1_000_000``. - -Mypy fully supports this syntax: - -.. code-block:: python - - precise_val = 1_000_000.000_000_1 - hexes: List[int] = [] - hexes.append(0x_FF_FF_FF_FF) - .. _async_generators_and_comprehensions: Asynchronous generators (`PEP 525 `_) and comprehensions (`PEP 530 `_) ---------------------------------------------------------------------------------------------------------------------------------------------------------- Python 3.6 allows coroutines defined with ``async def`` (PEP 492) to be -generators, i.e. contain ``yield`` expressions, and introduces a syntax for -asynchronous comprehensions. Mypy fully supports these features, for example: +generators, i.e. contain ``yield`` expressions. It also introduced a syntax for +asynchronous comprehensions. This example uses the ``AsyncIterator`` type to +define an async generator: .. code-block:: python @@ -93,4 +63,5 @@ asynchronous comprehensions. Mypy fully supports these features, for example: New named tuple syntax ---------------------- -Python 3.6 supports an alternative syntax for named tuples. See :ref:`named-tuples`. +Python 3.6 supports an alternative, class-based syntax for named tuples. +See :ref:`named-tuples` for the details. diff --git a/docs/source/supported_python_features.rst b/docs/source/supported_python_features.rst index ca68d4cf6b6b..20e01f096da7 100644 --- a/docs/source/supported_python_features.rst +++ b/docs/source/supported_python_features.rst @@ -1,5 +1,5 @@ -Supported Python features and modules -===================================== +Supported Python features +========================= A list of unsupported Python features is maintained in the mypy wiki: