ParamSpec can not express Function taking Function and it's arguments

[RunOrVeith]

Bug Report

Defining a type that says: "I am a function that takes a function and its arguments" does not get accepted by mypy.

To Reproduce

import contextlib
from concurrent.futures import Future, ThreadPoolExecutor

from typing import Callable, Iterator, ParamSpec, TypeVar

Signature = ParamSpec("Signature")
T = TypeVar("T")

TakesFunctionWithArguments = Callable[
    [Callable[Signature, T], Signature.args, Signature.kwargs],
    Future[T]
]


@contextlib.contextmanager
def submit_wrapper() -> Iterator[TakesFunctionWithArguments]:
    with ThreadPoolExecutor() as pool:
        def my_submit(func: Callable[Signature, T], *args: Signature.args, **kwargs: Signature.kwargs) -> Future[T]:
            return pool.submit(func, *args, **kwargs)

        yield my_submit


def foo(a: int, b: int, c: int) -> int:
    return a + b + c


with submit_wrapper() as submit:
    submit(foo, a=1, b=2, c=3)

Expected Behavior

This should type check correctly. It runs fine.

Actual Behavior

/home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:10: error: The first argument to Callable must be a list of types or "..."
/home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:10: error: Name "Signature.args" is not defined
/home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:10: error: Name "Signature.kwargs" is not defined
/home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:29: error: Unexpected keyword argument "a"
/home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:29: error: Unexpected keyword argument "b"
/home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:29: error: Unexpected keyword argument "c"

There are two errors here, I am not sure how related they are, so I have them both in this ticket:

1. The type alias on top is not accepted, because it can not relate the Signature ParamSpec within the Callable (I assume).
2. When calling the function, it does not detect that a,b,c are actually valid keyword arguments to foo.

Your Environment

• Mypy version used: mypy 0.950 (compiled: yes)
• Mypy command-line flags: None, but if you add --strict it adds the error /home/veith/.config/JetBrains/PyCharm2022.1/scratches/scratch_7.py:16: error: Missing type parameters for generic type "TakesFunctionWithArguments"
• Python version used: 3.10.4
• Operating system and version: Ubuntu 18.04

Possibly related to #12595

[erictraut]

Callable does not support *args or **kwargs parameters, so you cannot use P.args and P.kwargs within a Callable. You would need to use a callback protocol here instead.

Something like this:

class TakesFunctionWithArguments(Protocol):
    def __call__(
        self, func: Callable[P, T], *args: P.args, **kwargs: P.kwargs
    ) -> Future[T]:
        ...

Also, TakesFunctionWithArguments in your code sample is a generic type alias that accepts two type arguments, but when you use it, you are not providing any type arguments. That means mypy will assume Any for T and ... for Signature. That's probably not what you intend here.

Here's a modified sample that does type check correctly in mypy:

import contextlib
from concurrent.futures import Future, ThreadPoolExecutor
from typing import Callable, Iterator, ParamSpec, Protocol, TypeVar

P = ParamSpec("P")
T = TypeVar("T")

class TakesFunctionWithArguments(Protocol):
    def __call__(
        self, func: Callable[P, T], *args: P.args, **kwargs: P.kwargs
    ) -> Future[T]:
        ...

@contextlib.contextmanager
def submit_wrapper() -> Iterator[TakesFunctionWithArguments]:
    with ThreadPoolExecutor() as pool:

        def my_submit(
            func: Callable[P, T], *args: P.args, **kwargs: P.kwargs
        ) -> Future[T]:
            return pool.submit(func, *args, **kwargs)

        yield my_submit

def foo(a: int, b: int, c: int) -> int:
    return a + b + c

with submit_wrapper() as submit:
    submit(foo, a=1, b=2, c=3)

[AlexWaygood]

Exactly as Eric says: this kind of callable is inexpressible using the shorthand Callable[<params>, <return>] syntax. A callback protocol is the way to go.

There's some good mypy docs on callback protocols here: https://mypy.readthedocs.io/en/stable/protocols.html#callback-protocols