PEP 702: Type system support for deprecations

.github/CODEOWNERS

@@ -582,6 +582,7 @@ pep-0698.rst  @jellezijlstra
 pep-0699.rst  @Fidget-Spinner
 pep-0700.rst  @pfmoore
 pep-0701.rst  @pablogsal @isidentical @lysnikolaou
+pep-0702.rst  @jellezijlstra
 # ...
 # pep-0754.txt
 # ...

pep-0702.rst

@@ -0,0 +1,308 @@
+PEP: 702
+Title: Marking deprecations using the type system
+Author: Jelle Zijlstra <[email protected]>
+Status: Draft
+Type: Standards Track
+Topic: Typing
+Content-Type: text/x-rst
+Created: 30-Dec-2022
+Python-Version: 3.12
+
+
+Abstract
+========
+
+This PEP adds an ``@typing.deprecated()`` decorator that marks a class or function
+as deprecated, enabling static checkers to warn when it is used.
+
+Motivation
+==========
+
+As software evolves, new functionality is added and old functionality becomes
+obsolete. Library developers want to work towards removing obsolete code while
+giving their users time to migrate to new APIs. Python provides a mechanism for
+achieving these goals: the :exc:`DeprecationWarning` warning class, which is
+used to show warnings when deprecated functionality is used. This mechanism is
+widely used: as of the writing of this PEP, the CPython main branch contains
+about 150 distinct code paths that raise :exc:`!DeprecationWarning`. Many
+third-party libraries also use :exc:`!DeprecationWarning` to mark deprecations.
+In the `top 5000 PyPI packages <https://dev.to/hugovk/how-to-search-5000-python-projects-31gk>`__,
+there are:
+
+- 1911 matches for the regex ``warnings\.warn.*\bDeprecationWarning\b'``,
+  indicating use of :exc:`!DeprecationWarning` (not including cases where the
+  warning is split over multiple lines);
+- 1661 matches for the regex ``^\s*@deprecated``, indicating use of some sort of
+  deprecation decorator.
+
+However, the current mechanism is often insufficient to ensure that users
+of deprecated functionality update their code in time. For example, the
+removal of various long-deprecated :mod:`unittest` features had to be
+`reverted <https://github.com/python/cpython/commit/b50322d20337ca468f2070eedb051a16ee1eba94>`__
+from Python 3.11 to give users more time to update their code.
+Users may run their test suite with warnings disabled for practical reasons,
+or deprecations may be triggered in code paths that are not covered by tests.
+
+Providing more ways for users to find out about deprecated functionality
+can speed up the migration process. This PEP proposes to leverage static type
+checkers to communicate deprecations to users. Such checkers have a thorough
+semantic understanding of user code, enabling them to detect and report
+deprecations that a single ``grep`` invocation could not find. In addition, many type
+checkers integrate with IDEs, enabling users to see deprecation warnings
+right in their editors.
+
+Rationale
+=========
+
+At first glance, deprecations may not seem like a topic that type checkers should
+touch. After all, type checkers are concerned with checking whether code will
+work as is, not with potential future changes. However, the analysis that type
+checkers perform on code to find type errors is very similar to the analysis
+that would be needed to detect usage of many deprecations. Therefore, type
+checkers are well placed to find and report deprecations.
+
+Other languages already have similar functionality:
+
+* GCC supports a ``deprecated`` `attribute <https://gcc.gnu.org/onlinedocs/gcc-3.1.1/gcc/Type-Attributes.html>`__
+  on function declarations. This powers CPython's ``Py_DEPRECATED`` macro.
+* GraphQL `supports <https://spec.graphql.org/June2018/#sec-Field-Deprecation>`__
+  marking fields as ``@deprecated``.
+* Kotlin `supports <https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-deprecated/>`__
+  a ``Deprecated`` annotation.
+* Scala `supports <https://www.scala-lang.org/api/2.12.5/scala/deprecated.html>`__
+  an ``@deprecated`` annotation.
+* Swift `supports <https://docs.swift.org/swift-book/ReferenceManual/Attributes.html>`__
+  using the ``@available`` attribute to mark APIs as deprecated.
+* TypeScript `uses <https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html#deprecated>`__
+  the ``@deprecated`` JSDoc tag to issue a hint marking use of
+  deprecated functionality.
+
+Several users have requested support for such a feature:
+
+* `typing-sig thread <https://mail.python.org/archives/list/[email protected]/thread/E24WTMQUTGKPFKEXVCGGEFFMG7LDF3WT/>`__
+* `Pyright feature request <https://github.com/microsoft/pyright/discussions/2300>`__
+* `mypy feature request <https://github.com/python/mypy/issues/11439>`__
+
+
+Specification
+=============
+
+A new decorator ``@deprecated()`` is added to the :mod:`typing` module. This
+decorator can be used on a class, function or method to mark it as deprecated.
+This includes :class:`typing.TypedDict` and :class:`typing.NamedTuple` definitions.
+With overloaded functions, the decorator may be applied to individual overloads,
+indicating that the particular overload is deprecated. The decorator may also be
+applied to the overload implementation function, indicating that the entire function
+is deprecated.
+
+The decorator takes a single argument of type ``str``, which is a message that should
+be shown by the type checker when it encounters a usage of the decorated object.
+The message must be a string literal.
+The content of deprecation messages is up to the user, but it may include the version
+in which the deprecated object is to be removed, and information about suggested
+replacement APIs.
+
+Type checkers should produce a diagnostic whenever they encounter a usage of an
+object marked as deprecated. For deprecated overloads, this includes all calls
+that resolve to the deprecated overload.
+For deprecated classes and functions, this includes:
+
+* References through module, class, or instance attributes (``module.deprecated_object``,
+  ``module.SomeClass.deprecated_method``, ``module.SomeClass().deprecated_method``)
+* Any usage of deprecated objects in their defining module
+  (``x = deprecated_object()`` in ``module.py``)
+* If ``import *`` is used, usage of deprecated objects from the
+  module (``from module import *; x = deprecated_object()``)
+* ``from`` imports (``from module import deprecated_object``)
+
+There are some additional scenarios where deprecations could come into play:
+
+* An object implements a :class:`typing.Protocol`, but one of the methods
+  required for protocol compliance is deprecated.
+* A class uses the ``@override`` decorator from :pep:`698` to assert that
+  its method overrides a base class method, but the base class method is
+  deprecated.
+
+As these scenarios appear complex and relatively unlikely to come up in practice,
+this PEP does not mandate that type checkers detect them.
+
+Example
+-------
+
+As an example, consider this library stub named ``library.pyi``:
+
+.. code-block:: python
+
+   from typing import deprecated
+
+   @deprecated("Use Spam instead")
+   class Ham: ...
+
+   @deprecated("It is pining for the fiords")
+   def norwegian_gray(x: int) -> int: ...
+
+   @overload
+   @deprecated("Only str will be allowed")
+   def foo(x: int) -> str: ...
+   @overload
+   def foo(x: str) -> str: ...
+
+Here is how type checkers should handle usage of this library:
+
+.. code-block:: python
+
+   from library import Ham  # error: Use of deprecated class Ham. Use Spam instead.
+
+   import library
+
+   library.norwegian_gray(1)  # error: Use of deprecated function norwegian_gray. It is pining for the fiords.
+   map(library.norwegian_gray, [1, 2, 3])  # error: Use of deprecated function norwegian_gray. It is pining for the fiords.
+
+   library.foo(1)  # error: Use of deprecated overload for foo. Only str will be allowed.
+   library.foo("x")  # no error
+
+Runtime behavior
+----------------
+
+At runtime, the decorator sets an attribute ``__deprecated__`` on the decorated
+object. The value of the attribute is the message passed to the decorator.
+The decorator returns the original object. Notably, it does not issue a runtime
+:exc:`DeprecationWarning`.
+
+For compatibility with :func:`typing.get_overloads`, the ``@deprecated``
+decorator should be placed after the ``@overload`` decorator.
+
+Type checker behavior
+---------------------
+
+This PEP does not specify exactly how type checkers should present deprecation
+diagnostics to their users. However, some users (e.g., application developers
+targeting only a specific version of Python) may not care about deprecations,
+while others (e.g., library developers who want their library to remain
+compatible with future versions of Python) would want to catch any use of
+deprecated functionality in their CI pipeline. Therefore, it is recommended
+that type checkers provide configuration options that cover both use cases.
+As with any other type checker error, it is also possible to ignore deprecations
+using ``# type: ignore`` comments.
+
+Deprecation policy
+------------------
+
+We propose that CPython's deprecation policy (:pep:`387`) is updated to require that new deprecations
+use the functionality in this PEP to alert users
+about the deprecation, if possible. Concretely, this means that new
+deprecations should be accompanied by a change to the ``typeshed`` repo to
+add the ``@deprecated`` decorator in the appropriate place.
+This requirement does not apply to deprecations that cannot be expressed
+using this PEP's functionality.
+
+Backwards compatibility
+=======================
+
+Creating a new decorator poses no backwards compatibility concerns.
+As with all new typing functionality, the ``@deprecated`` decorator
+will be added to the ``typing_extensions`` module, enabling its use
+in older versions of Python.
+
+How to teach this
+=================
+
+For users who encounter deprecation warnings in their IDE or type
+checker output, the messages they receive should be clear and self-explanatory.
+Usage of the ``@deprecated`` decorator will be an advanced feature
+mostly relevant to library authors. The decorator should be mentioned
+in relevant documentation (e.g., :pep:`387` and the :exc:`DeprecationWarning`
+documentation) as an additional way to mark deprecated functionality.
+
+Reference implementation
+========================
+
+A runtime implementation of the ``@deprecated`` decorator is
+`available <https://github.com/python/typing_extensions/pull/105>`__.
+The ``pyanalyze`` type checker has
+`prototype support <https://github.com/quora/pyanalyze/pull/578>`__
+for emitting deprecation errors.
+
+Rejected ideas
+==============
+
+Runtime warnings
+----------------
+
+Users might expect usage of the ``@deprecated`` decorator to issue a
+:exc:`DeprecationWarning` at runtime. However, this would raise a number of
+thorny issues:
+
+* When the decorator is applied to a class or an overload, the warning
+  would not be raised as expected.
+* Users may want to control the ``warn`` call in more detail (e.g.,
+  changing the warning class).
+* ``typing.py`` generally aims to avoid affecting runtime behavior.
+
+Users who want to use ``@deprecated`` while also issuing a runtime warning
+can use the ``if TYPE_CHECKING:`` idiom, for example:
+
+.. code-block:: python
+
+   from typing import TYPE_CHECKING
+   import functools
+   import warnings
+
+   if TYPE_CHECKING:
+       from typing import deprecated
+   else:
+       def deprecated(msg):
+           def decorator(func):
+               @functools.wraps(func)
+               def wrapper(*args, **kwargs):
+                   warnings.warn(msg, DeprecationWarning, stacklevel=2)
+                   return func(*args, **kwargs)
+               wrapper.__deprecated__ = msg
+               return wrapper
+           return decorator
+
+Deprecation of modules and attributes
+-------------------------------------
+
+This PEP covers deprecations of classes, functions and overloads. This
+allows type checkers to detect many but not all possible deprecations.
+To evaluate whether additional functionality would be worthwhile, I
+`examined <https://gist.github.com/JelleZijlstra/ff459edc5ff0918e22b56740bb28eb8b>`__
+all current deprecations in the CPython standard library.
+
+I found:
+
+* 74 deprecations of functions, methods and classes (supported by this PEP)
+* 28 deprecations of whole modules (largely due to :pep:`594`)
+* 9 deprecations of function parameters (supported by this PEP through
+  decorating overloads)
+* 1 deprecation of a constant
+* 38 deprecations that are not easily detectable in the type system (for
+  example, for calling :func:`asyncio.get_event_loop` without an active
+  event loop)
+
+Modules could be marked as deprecated by adding a ``__deprecated__``
+module-level constant. However, the need for this is limited, and it
+is relatively easy to detect usage of deprecated modules simply by
+grepping. Therefore, this PEP omits support for whole-module deprecations.
+As a workaround, users could mark all module-level classes and functions
+with ``@deprecated``.
+
+For deprecating module-level constants, object attributes, and function
+parameters, a ``Deprecated[type, message]`` type modifier, similar to
+``Annotated`` could be added. However, this would create a new place
+in the type system where strings are just strings, not forward references,
+complicating the implementation of type checkers. In addition, my data
+show that this feature is not commonly needed.
+
+Acknowledgments
+===============
+
+A call with the typing-sig meetup group led to useful feedback on this
+proposal.
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.