Overconstrained type variable prevents declaring Protocols with shared invariant type

[butlscot]

From #5775 I learned that

mypy enforces that when the TypeVar can be covariant, it must be covariant

This decision seems to prevent users from declaring Protocols that share an invariant generic type.

For a simple practical example, a depth-first search framework:

from typing import TypeVar, Protocol


NodePassthrough = TypeVar("NodePassthrough")


class Node(Protocol):
    def iter(self) -> list["Node"]:
        """Returns a list of child nodes."""
        ...


class DfsEnterNodeFn(Protocol[NodePassthrough]):
    def __call__(self, node: Node) -> NodePassthrough:
        """Called when entering a node during depth-first traversal."""
        ...


class DfsLeaveNodeFn(Protocol[NodePassthrough]):
    def __call__(self, node: Node, node_passthrough: NodePassthrough) -> None:
        """Called before leaving a node during depth-first traversal."""
        ...


def dfs(node: Node, upon_entering: DfsEnterNodeFn[NodePassthrough], before_leaving: DfsLeaveNodeFn[NodePassthrough]) -> None:
    """Depth-first search of a tree of nodes, calling upon_entering and before_leaving for each node encountered."""
    node_passthrough = upon_entering(node)

    for child in node.iter():
        dfs(child, upon_entering, before_leaving)

    before_leaving(node, node_passthrough)

This results in the following mypy errors:

example.py:12: error:
Invariant type variable "NodePassthrough" used in protocol where covariant one is expected  [misc]
    class DfsEnterNodeFn(Protocol[NodePassthrough]):
    ^

example.py:18: error:
Invariant type variable "NodePassthrough" used in protocol where contravariant one is expected  [misc]
    class DfsLeaveNodeFn(Protocol[NodePassthrough]):
    ^

Is there a workaround for this?

[JelleZijlstra]

Yes, you can use different type variables.

"Sharing" type variables is a nebulous concept anyway. From the type checker's perspective, every generic class's type variables are completely independent. For this reason, the native generic syntax introduced in Python 3.12 makes it so every type variable is declared separately per generic class or function.

[butlscot]

Ah I see what you mean from PEP 695.

The scoping rules for type variables are difficult to understand. Type variables are typically allocated within the global scope, but their semantic meaning is valid only when used within the context of a generic class, function, or type alias. A single runtime instance of a type variable may be reused in multiple generic contexts, and it has a different semantic meaning in each of these contexts.

Python's typing system itself does not support defining type variables that capture the constraints (explicit or inferred) between two different generic contexts, and enforce the set of shared constraints in both.

Do you think it could be useful for mypy to detect these kinds of mistakes and explicitly call attention to the issue? For example, recognizing that I am reusing a single TypeVar in two generic contexts with incompatible variance constraints, it could give an additional error calling attention to this common misconception.