Annotate decorated functions

[sixolet]

When you declare a function, your signature annotation serves three purposes.

• It allows a program to typecheck the body of the function, finding human errors
• It allows a program to typecheck the call sites of the function, finding more human errors
• It allows a human to quickly understand the usage of the function, preventing human errors.

Please excuse my poorly-motivated example below.

from typing import TypeVar, Callable, Dict, Any

T = TypeVar('T')

def reprify(f: Callable[[T], Any]) -> Callable[[T], str]:
    def res(__x: T) -> str:
        return repr(f(__x))
    return res

@reprify
def nonsense(i: int) -> Dict[str, int]:
    return {"nonsense": i}

s = nonsense(3)
reveal_type(s)  # E: Revealed type is 'builtins.str'

The first two purposes are served. The computer can typecheck the body of the function, understanding that it is to return a dictionary inside. The computer can also typecheck the call site of the function, understanding that the decorator causes the function to return a string.

The last purpose is not served. To understand how to use the nonsense function, a human must look at the signature of nonsense, then remember to look at the signature of reprify, and then perform a type calculation in their head combining the two.

I would like a way to optionally make a description of the type of nonsense a call site sees, so the humans do not get confused. We can then tell a computer to check against the decorators.

One possibility, provided for its straw-person properties:

...

@reprify  # type: (int) -> str
def nonsense(i: int) -> Dict[str, int]:
    return {"nonsense": i}

...

[gvanrossum]

Hm, this has always bothered me too (though my solution is to avoid decorators like that unless they have a momentous benefit, and then to just memorize the transmogrification represented by the decorator and apply it in my head when reading the code).

I like your straw-individual too.

[JukkaL]

I've occasionally wanted a feature like this. This would also let users manually annotate a function when the decorator doesn't have a precise signature, resulting in a Callable[..., Any] signature (or similar) otherwise.

[gvanrossum]

Let's do this. We need to (a) make a PR for PEP 484, discuss it briefly, and I'll accept it; (b) make changes to typed_ast to support type comments in these syntactic positions; (c) make changes in mypy to use it.

We'd also need to get PyCharm ( @vlasovskikh ) and PyType ( @matthiaskramm ) to be on board, that's the most important goal of (a) above.

[matthiaskramm]

+1 from the pytype side. We definitely had that "imprecise decorator" problem, especially when automatically generating a pyi for the module containing the decorator.

Would we do this for typeshed (i.e, in pyi stubs), as well? Looking through the repo, the primary use case seems to be functions decorated with @contextmanager.

[gvanrossum]

mypy would definitely support this everywhere.

[sixolet]

I'll happily write a PR against PEP484, but I have one more question before getting there, regarding a hole I'm finding in the syntax proposal I made:

What happens when the eventual function signature is too long? For normal functions, we have the one-per-line type-comment syntax, but that doesn't seem possible here. Can we, or should we, pair this with a syntax for continuation-lines for type comments? Is there another possible solution that does not have this problem?

[gvanrossum]

Maybe allow a callable type alias too?

[gvanrossum]

E.g.

TT = Callable[[int], str]
@reprify # type: TT
def nonsense(i: int) -> Dict[str, int]:
    return {"nonsense": i}

[sixolet]

Yesssssss that solves that problem.

Maybe we should do one of these things too:

• Annotating a decorator line makes the relevant object be exactly the type of the annotation -- no special function (blah) -> blag case needed. If you have an annotation that takes in a function and spits out a ContextManager, then that's what the type of the resulting thing will be?
• Make a Callable or alias thereof be allowed in the # type: (blah) -> blag comment inside the function, for consistency?

[gvanrossum]

Annotating a decorator line makes the relevant object be exactly the type of the annotation [...]

This is a good point. There's no rule that says a decorator needs to return a function or even a Callable. So maybe we shouldn't allow # type: (argtypes) -> rtype at all (it's easier if that syntax is exclusive to def anyway).

Make a Callable or alias thereof be allowed in the # type: (blah) -> blag comment inside the function, for consistency?

I'd rather not go there. So the revised proposal is that only a "regular" type comment should be allowed on the @decorator line. I like it even better! (And hadn't appreciate this aspect of the proposal before.)

[sixolet]

Sounds great. I'll go for writing up the version where you can add a #type: blah annotation to a decorated function or class, and it declares that the type will be blah after passing through the decorator.

[vlasovskikh]

@sixolet @gvanrossum The idea looks good. I'll join the PEP review.

[sixolet]

From the PEP review -- we landed on a new decorator, instead of a type comment.

[gpshead]

The ability to annotate decorated functions in a human readable and writable also matters for coroutine based systems.

Using a Google App Engine ndb.tasklet API as an example, I believe it would look something like this using the current proposals:

@decorator_type(Future[str])
@ndb.tasklet
def Example() -> ???:
    foo, bar  = yield fetch_foo(), fetch_bar()
    raise ndb.Return('foo: {}, bar: {}'.format(foo, bar))

Without support for decorator annotations the Example return value might be an entirely gross looking:

@ndb.tasklet
def Example() -> Generator[Sequence[Future[Union[FooClass, BarClass]]]]:

@bslatkin can probably give better more realistic examples of that than I can.

This probably applies equally to decorator based APIs such as to @asyncio.coroutine as well, though I'm not up to date on what the more modern async def syntax means in that regard.