Need a way to refer to the type parameter of a generic class inside a generic function

[pkch]

I don't think there's a way to write this code in a type-safe way:

from typing import TypeVar, Generic, List, Callable
T = TypeVar('T')
class X(Generic[T]):
    def f(self) -> T: ...

C = TypeVar('C', bound=X)

def g(cls: Callable[[], C]) -> None:
    x: ??? = cls().f()

There is no precise type I can use in place of ???.

reveal_type(cls().f()) tells me Revealed type is 'T`1', but that's not much help because I can't say x: T (type arguments of a generic function are determined only by the signature, so T would be invalid in that context).

Nor can I change C to become TypeVar('C', bound=X[T]) (can't use type arguments in that context either).

Nor can I use def g(cls: Callable[[], C[T]]) -> None (type variables cannot be used with arguments).

So ??? is a type that mypy understands, but there's no way to actually refer to it.

Of course, the toy example above works without annotation (mypy simply infers type of x) but if I need to make a list of those things, I'm out of luck:

    def g(cls: Callable[[], C]) -> None:
        x: List[???] = []
        for i in range(10):
            x.append(cls().f())

[ilevkivskyi]

If I understand your example correctly, then this could be expressed by higher kinds (e.g. you want your C variable to be * -> *). But this is quite advanced feature, and I am not sure mypy will or even should support this.

Why exactly do you need variable C and not just Callable[[], X[T]]? Subtypes of X will also be accepted there, for example:

class X(Generic[T]):
    def f(self) -> T: ...

class Y(X[T]): ...

def g(cls: Callable[[], X[T]]) -> T:
    x: List[T] = []
    x.append(cls().f())
    return x[0]

reveal_type(g(Y[str]))  # Revealed type is 'builtins.str*'

[pkch]

The reason I need C rather than X[T] is that in my actual code, the return type of g should also be C. Here's a slightly more precise reproduction of what I have. Let's say I'm implementing a graph data structure (edited for better context):

from typing import Generic, Optional, TypeVar, Set, Callable, DefaultDict
from collections import defaultdict

T = TypeVar('T')  # user-provided values, which we'll wrap in class Node

class Node(Generic[T]):
    value: Optional[T]
    ...


# ABC that describes interface for Graph
class Graph(Generic[T]):
    nodes: Set[Node[T]]
    def add_node(self) -> Node[T]: ...
    ...


class MyGraph(Graph[T]):
    def add_node(self) -> Node[T]:
        n: Node[T] = Node()
        ...
        return n


G = TypeVar('G', bound=Graph)  # specific implementation of Graph ABC

# this function works fine with any implementation of Graph
def read_graph(cls: Callable[[], G], s: str) -> G:
    g = cls()
    nodes: DefaultDict[str, Node[???]] = defaultdict(g.add_node)
    ...
    return g

read_graph(cls=MyGraph, s='...')

I can't use

def read_graph(cls: Callable[[], Graph[T]], s: str) -> Graph[T]:
    ...

because if read_graph() is called with cls = MyGraph, the return type should be MyGraph rather than just the abstract base class Graph.

As to higher kinds, I never heard about them until now, but from what I could understand at first glance, you're right, this is one way to achieve this. But, consistent with your comments, it is slightly more advanced (it's in Scala/Haskell AFAIU, but not in C#/Java).

I can see that due to complexity (conceptual and implementation) it may not be relistic to add it to python type hints, at least not soon. Maybe there's a simpler solution for my immediate use case?

[gvanrossum]

I don't understand what you're trying to do here. The read_graph() function you show in your last comment is not generic, since the occurrences of Graph without following [] are interpreted (by PEP 484 and by mypy) as using Graph[Any]. Did you mean to write Graph[G]? Your argument starting with "I can't use Graph[T] here because..." sounds backwards, which probably means I don't understand it.

I'd also like to see more of your implementation of Node -- why isn't it generic?

---

Also, honestly, this is beginning to look like you may be so in love with generics you want to use them everywhere, and the code you're writing is like "writing Scala/Haskell in Python" (like there used to be a saying "you can write FORTRAN in any language" :-).

I recommend taking a step back -- maybe you don't need everything to be so generic/general? Remember that we're trying to keep the type system so that it is easily taught to Python programmers, not to satisfy people with a Ph.D in functional programming. :-)

[pkch]

@gvanrossum Sorry, my example had a typo. I edited my previous comment to fix it and also to include a better reproduction of the context.

Regarding Graph[T], I was responding to @ilevkivskyi suggestion to get rid of type variable entirely, and replace it with the parameterized class. It might be a good solution sometimes, but not in this context:

def read_graph(cls: Callable[[], Graph[T]], s: str) -> Graph[T]:
    ...

won't work because read_graph(cls=MyGraph, s='...') should have return type MyGraph[T] rather than Graph[T].

[gvanrossum]

OK, now I see. You have a generic function over a type variable G that represents some Graph[T] and you need to reference Node[T] in the function. Pragmatically you should just use Node[Any] and move on. The redesign of PEP 484 needed to support this in a more type-safe way doesn't sound like an easy thing, and we really have a lot of more important things to do first (e.g. we've had to delay work on the Protocols PR because the Dropboxers that can review it all have too much else on their TODO list already).

[pkch]

Yes, I just didn't realize how easy it is to end up with more complex types.

I'm perfectly ok with the type system that has some limitations, but can be easily overridden with Any or type: ignore.

[pkch]

This entire issue disappears (in a sense) once I realized that type variables cannot be bound by generic types with free type parameters as confirmed in #2756. Once #3153 is fixed, mypy will correctly interpret

C = TypeVar('C', bound=X)

as

C = TypeVar('C', bound=X[Any])

and with that, it will be obvious that the only way to annotate x in my original example is as Any.

FWIW, in some cases, there's a workaround: make X non-generic in the first place, and instead use generic methods:

from typing import TypeVar, Generic, List, Callable
T = TypeVar('T')
class X:
    def f(self: T) -> T: ...

C = TypeVar('C', bound=X)

def g(cls: Type[C]) -> None:
    x: C = cls().f()

If this is not enough (e.g., because X needs generic attributes, etc.), then we've hit the understandable limits of the type system.