Support Callable[..., t] (with literal ellipsis, unspecified argument types and specified return type)

[abarnert]

It would be handy to be able to specify a callable whose argument types are not checked but whose return type is, e.g., by specifying the argument types as None instead of a list.

For example, consider map:

def map(function, *iterables):

How would you annotate that? Trying to actually specify the argument types to function to match the element types of the iterables in iterables would require something horrible like C++11's parameter packs and variable-length type tuples. But just specifying Function means there's no way to know that the result iterates values of the same type the function returns, which seems like a pity.

If you could just use None or Any instead of a list, you could do this:

def map(function: Function[None, T], *iterables: Iterable[Iterable]) -> Iterator[T]:

[abarnert]

I notice that in the existing stubs you've tried to annotate map like this:

# TODO more than two iterables
@overload
def map(func: Function[[T1], S], iter1: Iterable[T1]) -> Iterator[S]: pass
@overload
def map(func: Function[[T1, T2], S],
        iter1: Iterable[T1],
        iter2: Iterable[T2]) -> Iterator[S]: pass

This is similar to the way people used to do things in C++03. Boost even came with a preprocessor library that helped you generate 10 overloads for your template, with 0 through 9 parameters. But this was a huge mess. And, as I noted above, I don't really think we want the C++11 solution (which would require using a template parameter pack to capture the types of iterable, then forwarding through a recursive function template to extract the T out of each Iterable[T], and another recursive function template to pack them back up into a single type tuple).

I think the easy answer is just to not try to fully annotate all possibilities. Yes, it means that map((lambda x, y, z: x+y+z), file1, file2, range(100)) would pass the type check even though it shouldn't, but how often is that going to come up?

[JukkaL]

You can just leave out types of (some) arguments or give them Any types (which means the same thing). Variables with Any types are dynamically typed and are compatible with everything. For example:

from typing import Any
def f(x: Any) -> int: return 1
f(1)  # Ok
f('x')  # Ok
f(1) + 'x'  # Not ok, can't add int and str

[JukkaL]

My gut feeling is that it makes sense to have ugly annotations for map and zip since they are in builtins and they are used pretty often in the wild. Also, by not properly annotating the arguments the return values would have imprecise types, which would propagate elsewhere via type inference, and this would be a shame.

We should probably have a fallback overload variant that takes *args to cover all cases (even if this would imply less precise types).

However, for less frequently used modules/functions falling back to dynamic typing is probably the most reasonable approach. I.e., we'd consider map/zip as exceptions/legacy that we just need to be able to type check properly, even if it's hacky or does not work in every case.

Another argument is that one the first examples that a programmer with a functional programming background tries to type check uses map, and we'd give a pretty poor first impression if we couldn't type check that... :-)

[abarnert]

You can just leave out types of (some) arguments or give them Any types (which means the same thing).

You're missing the point here. (Probably my fault.) I'm asking how to declare the types of arguments (or return values) that are themselves functions.

This is why I used map as an example, because it's the most familiar higher-order function. Its first argument is a function. The argument list of that function is dependent on the other arguments to map. That makes it very hard to specify.

Also, by not properly annotating the arguments the return values would have imprecise types, which would propagate elsewhere via type inference, and this would be a shame.

That's exactly my point. Sometimes, it's hard, verbose, or even impossible to annotate the argument types of the function argument, but easy to annotate the return type of the function argument. And often that's sufficient to provide a precise return type for the higher-order function.

Again, look at map. If all I know is that its first argument is a Function[???, int], that's enough to tell me that its return type is int.

So, it's useful to be able to specify an argument type that means Function[???, int] in some way, which I suggested spelling Function[None, int].

[JukkaL]

Ah, sorry, I misunderstood your point. There is currently no way to do this. I guess we could have extra syntax for something like this, such as Function[[...], int], though maybe that would be abusing '...' (and it wouldn't work in Python 2).

More generally, callable types are restricted to a fixed number of arguments and there's no way to have keyword arguments, *args*, **kwargs, or arguments with default values in callable types. This would be technically simple but it's tricky to come up with a reasonable syntax.

Here are some (maybe stupid) ideas:

Function[[int, [str]], None]  # default arg value
Function[[int, vararg(int)], None]  # *args
Function[args(str, x=int, *int), None]  # default value (usable as keyword arg) and *args
Function[[('x=', 'int), ('*', int)], None]  # default arg value (usable as keyword arg) and *args

[abarnert]

I wasn't even trying to solve the fully general problem, because I was assuming that it's unsolvable, or at least not worth solving. If you do want to solve it, it's even worse than you're thinking. Because of keyword arguments, even the parameter names can matter, not just the number and types. So you need something at least as complex as inspect.Signature and inspect.ArgSpec to determine whether a function is actually callable with a set of arguments. But that also points at the possible solution: For anything more complicated than a fixed list of arguments, just take an inspect.Signature. (I'm not sure even that's sufficient, but if not, it would be easier to build from there than to start from scratch and try to cover all the cases that it's had to tackle that you're probably not thinking of yet…)

Getting this right is part of why programming in Swift, or doing compile-time metaprogramming in C++, is not as much fun as programming in Python. ML and Haskell solve this by just not allowing anything fancy. In Haskell, all functions take one argument; if you need more, you either take a tuple, or you write it curried (which are of course equivalent). That makes rigorously defining the type of any function dead simple. Too bad that won't work for Python (or C++ or Swift).

But maybe the complicated cases don't need to be solved, as long as the simple ones can be.

I think my proposal of just using None, or maybe Any, in place of the list of arguments is good enough for the case I'm worried about:

Function[None, int] # function that takes who-knows-what but definitely returns int

There's one intermediate case that seems potentially important: I want a function that takes an int and a str. It may take other positional or keyword-only parameters with default values; it may have *args or **kwargs; it may actually only take an int and a *args that will accept my str; I don't care, as long as I can call it with an int and a str. Is that what Function[[int, str], None] means, or does that mean specifically a function that must take an int and a str and nothing else?

[JukkaL]

Function[[int, str], None] means a function that can be called with two positional arguments, the first with type int and the second with type str.

The current syntax seems to cover the majority of use cases for callable values. I was aware of the potential complexity and that's why the current annotation syntax is so simplistic.

Another option would be to use Function[Any, int] for a function that is compatible with arbitrary argument lists but that must return int. I actually kinda like this, though it could be potentially confusing.

map is tricky because the function is often a lambda, and we need to infer the return type. Additionally, the return type often depends on the argument type(s), which must be inferred as well.

[JukkaL]

The PEP 484 draft uses Callable[..., T] (with literal ...) for a callable type with arbitrary arguments but with a specific return type T. Mypy should support that.