TypeVars can't be assigned to other TypeVars

[SKHecht]

I've found a few cases of mypy throwing errors saying that two TypeVars are incompatible instead of simply assigning one TypeVar to another. Here's the simplest example:

from typing import TypeVar
S = TypeVar('S')
T = TypeVar('T')
def foo(x: S) -> T:
    return x

mypy says:

m.py:5: error: Incompatible return value type (got "S", expected "T")

The error doesn't seem to appear if any of the TypeVars gets assigned a concrete type. mypy has no issues with this file:

from typing import Callable, TypeVar
T = TypeVar('T')
S = TypeVar('S')
def format(x: T, formatter: Callable[[T], S]) -> S:
    return formatter(x)
format(0, lambda y: y)

However, if I make the lambda a default value:

from typing import Callable, TypeVar
T = TypeVar('T')
S = TypeVar('S')
def format(x: T, formatter: Callable[[T], S] = lambda y: y) -> S:
    return formatter(x)
format(0)

mypy gives this error

n.py:4: error: Incompatible default for argument "formatter" (default has type "Callable[[T], T]", argument has type "Callable[[T], S]")
n.py:4: error: Incompatible return value type (got "T", expected "S")

I suspect this is related to #4814

[ilevkivskyi]

The first example is not a bug but expected behavior. Also inability to specify a default for a generic argument is a known issue, see #3737

In your case however, mypy is correct, there is no way it can infer S in a single-argument call to format(). Instead you should use an overload:

@overload
def format(x: T) -> T: ...
@overload
def format(x: T, formatter: Callable[[T], S]) -> S: ...
def format(x, formatter = lambda y: y):
    # your implementation here
    ...

[SKHecht]

@ilevkivskyi would you mind expanding on why the first example is expected behavior? If there exists an assignment to the variables such that the code type checks, shouldn't the code be valid? If such an assignment leaves free variables, then the expression remains generic.

Both assignments T <- S and S <- T would cause the code to properly type check, and leave either S or T (resp) a free variable. Admittedly, this adds some nondeterminism in the behavior of mypy, but only in the case of code that doesn't type properly, ie, if you did

x: int = 0
y: str = foo(x)

the error could either be with the argument x or the assignment to y. But either error would work, and code that does type check would not have this issue.

Given that my first example is expected behavior, I can understand why my third example wouldn't work (even if #3737 was fixed). However, if something like my first example did type check (and the bug was fixed), then what would be the difference between the second and third example?

One way to resolve format(0, lambda y: y) with Callable[[T, Callable[[T], S]], S] is (I don't know mypy's resolution algorithm, but this is my guess as to how it runs):

• resolve arg list 0, lambda y: y with type list T, Callable[[T], S]
  1. resolve 0 with T, thus T <- int
  2. resolve lambda y: y with Callable[[T], S], given { T <- int }
     a. resolve lambda y: y with Callable[[int], S]
     b. lambda y: y <- Callable[[int], ?]
     c. lambda y: y <- Callable[[int], int], thus S <- int
  3. type of arg list is int, Callable[[int], int], given { T <- int, S <- int }
• return type is S, given { T <- int, S <- int }
• return type is int

This works with the current state of mypy because the type of x is concrete.

However, if we allow for type variables to be assigned to other type variables, then the resolution for the single argument format (at its definition) could be similar:

• resolve arg list x, lambda y: y with type list T, Callable[[T], S], given x <- T
  1. first arg is of type T, not further resolution possible, T is free variable
  2. resolve lambda y: y with Callable[[T], S]
     a. resolve lambda y: y with Callable[[T], S], given T is a free variable
     b. lambda y: y <- Callable[[T], ?]
     c. lambda y: y <- Callable[[T], T], thus S <- T
  3. type of arg list is T, Callable[[T], T], given { S <- T }
• return type is S, given { S <- T }
• type of single argument format is Callable[[T], T]

This resolution could have happened in the opposite order and resulted in the type Callable[[S], S], but both are logically consistent. Either way, format(0) types to int.

[ilevkivskyi]

Default values, and moreover function bodies are not part of the function API. Function type signatures form rigid interfaces between different namespaces and are not subject to inference.

In your first example foo might live in some library, while code calling it like x: str = foo(0) in some user code. In your second example, what would you do if you just have a stub file:

def format(x: T, formatter: Callable[[T], S] = ...) -> S: ...

You can't infer format(0).

[SKHecht]

Interesting, that makes sense and definitely explains the expected behavior. Seems like supporting the feature I was requesting would require a fundamental rewrite of mypy's internals.

Thanks for the informative and quick replies