what's the correct type for types.coroutine decorated generators?

[toejough]

I cannot seem to get a types.coroutine decorated generator to typecheck correctly.

I checked the tests out, and they seem to indicate that the right thing to do is to annotate the return as "typing.Generator", but when I do that, mypy complains that I cannot await that.

coro_example.py:

"""Coro typing example."""


import types
import typing


async def await_decorated() -> None:
    """Await a decorated generator."""
    x = await decorated_generator()


@types.coroutine
def decorated_generator() -> typing.Generator:
    """Yield None."""
    yield None

> mypy coro_example.py
coro_example.py:10: error: Incompatible types in await (actual type Generator[Any, Any, Any], expected type "Awaitable")

I thought perhaps I was subtly mistyping something, so I copy/pasted the code from the tests (from line 346 to line 403) and ran that, and got similar errors:

> mypy mypy_example.py
mypy_example.py:45: error: "yield from" can't be applied to Awaitable[int]
mypy_example.py:47: error: "yield from" can't be applied to AwaitableGenerator[Any, Any, int, Awaitable[int]]
mypy_example.py:49: error: "yield from" can't be applied to "Aw"
mypy_example.py:53: error: Incompatible types in await (actual type Generator[str, None, int], expected type "Awaitable")
mypy_example.py:57: error: Incompatible types in await (actual type "It", expected type "Awaitable")

If I type the generator as a typing.Awaitable instead, mypy tells me I have to type the generator as a generator or one of its supertypes.

I've tried this with python 3.6 and the latest github and pip versions:

• python 3.6.0
• mypy mypy (0.480.dev0) (installed via pip install git+https://github.com/python/mypy)
• mypy mypy (0.471) (installed via pip install mypy)

Am I doing it wrong, or is there a bug?

[ddfisher]

I believe you need to give it the type typing.AwaitableGenerator. It sounds like we may need to document this better somewhere.

[toejough]

yep, that seems to do the trick. thank you very much!

This type isn't presently documented for python 3.6's typing lib, and it doesn't show up in that lib's static attributes, either:

python
> python
Python 3.6.0 (default, Dec 23 2016, 14:42:08)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import typing
>>> dir(typing)
['AbstractSet', 'Any', 'AnyStr', 'AsyncIterable', 'AsyncIterator', 'Awaitable', 'BinaryIO', 'ByteString', 'CT_co', 'Callable', 'CallableMeta', 'ClassVar', 'Collection', 'Container', 'ContextManager', 'Coroutine', 'DefaultDict', 'Dict', 'FrozenSet', 'Generator', 'Generic', 'GenericMeta', 'Hashable', 'IO', 'ItemsView', 'Iterable', 'Iterator', 'KT', 'KeysView', 'List', 'Mapping', 'MappingView', 'Match', 'MutableMapping', 'MutableSequence', 'MutableSet', 'NamedTuple', 'NamedTupleMeta', 'NewType', 'Optional', 'Pattern', 'Reversible', 'Sequence', 'Set', 'Sized', 'SupportsAbs', 'SupportsBytes', 'SupportsComplex', 'SupportsFloat', 'SupportsInt', 'SupportsRound', 'T', 'TYPE_CHECKING', 'T_co', 'T_contra', 'Text', 'TextIO', 'Tuple', 'TupleMeta', 'Type', 'TypeVar', 'TypingMeta', 'Union', 'VT', 'VT_co', 'V_co', 'ValuesView', '_Any', '_ClassVar', '_FinalTypingBase', '_ForwardRef', '_G_base', '_Optional', '_PY36', '_Protocol', '_ProtocolMeta', '_TypeAlias', '_TypingBase', '_TypingEllipsis', '_TypingEmpty', '_Union', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '_check_generic', '_cleanups', '_eval_type', '_generic_new', '_geqv', '_get_defaults', '_get_type_vars', '_gorg', '_make_nmtuple', '_make_subclasshook', '_next_in_mro', '_no_slots_copy', '_overload_dummy', '_qualname', '_remove_dups_flatten', '_replace_arg', '_subs_tree', '_tp_cache', '_trim_name', '_type_check', '_type_repr', '_type_vars', '_valid_for_check', 'abc', 'abstractmethod', 'abstractproperty', 'cast', 'collections', 'collections_abc', 'contextlib', 'functools', 'get_type_hints', 'io', 'no_type_check', 'no_type_check_decorator', 'overload', 're', 'stdlib_re', 'sys', 'types']
>

Should I open a new issue with python directly?

[ddfisher]

Yeah, typing.AwaitableGenerator doesn't actually exist at runtime. I'm not sure what the plan for that is, exactly (and I don't see it documented in any issue currently). @gvanrossum, is AwaitableGenerator going to exist at runtime at some point?

[gvanrossum]

No, it's a crutch to make up for the lack of duck typing in mypy.

[toejough]

Is there a preferred place/format for me to contribute some documentation? I always think I'll make time to contribute some code to mypy, but I never end up doing so, but I could probably type up some quick explanation of this type.

[toejough]

I started a PR here with an example I found in the documentation updated and a little explanatory note added. Happy to add more or do it differently.

[gvanrossum]

Hm, the root cause of your original example is a forward reference to a decorated function, not anything having to do with Generator vs. AwaitableGenerator. When I simply swap the two functions the example passes without failure. This is unfortunate -- mypy has a subtle bug where decorated functions aren't fully processed until later, and one of the things that happens "later" is flagging functions decorated with @coroutine.

All the magic processing for AwaitableGenerator lives in mypy/checker.py; it's pretty magical and not intended to be used by end-user code. See e.g. this comment. The changing of the type to AwaitableGenerator is happening here; it uses the flag is_awaitable_coroutine which is set here.

So, apart from the forward reference issue, the intention is really that a function decorated with coroutine should be typed as Generator or one of its supertypes (i.e. Iterator, Iterable, object). It so happens that AwaitableGenerator also works, but you should not use it, and it is intentionally undocumented.

[ddfisher]

Oops -- that's good to know! I won't tell people about it, then. >_>

[toejough]

ho boy - well, thanks for the analysis and explanation! I really appreciate it, and I'm happy to have a way to move forward.

Is the forward reference bug in the code in mpy/checker.py, as well? If nothing else, I'd like to understand it better.

[gvanrossum]

NP. The forward reference issue is emergent. Basically the code in that sets the return type of the @coroutine-decorated function (checker.py line 551) runs after the code that uses it has already complained (checkexpr.py at line 2111). This is because mypy checks one function at a time, in definition order. There's an escape hatch for forward references (search checker.py for DeferredNode) but apparently it's not used here.

[natelust]

I recently ran into a problem similar to this, but a little more devious, and so none of the suggestions could work for me.

I have a function that wraps a callable inside a Coroutine and returns the generated Coroutine. A simplified version looks like:

def _makeCoroutine(func: Callable[str, str], executor: ThreadPoolExecutor, depends: Set[str]) ->\
        Callable[[str, str], Coroutine[Set[str], str, str]]:
    @types.coroutine
    def wrapper(a: str, b: str)-> str:
        executor.submit(func, a, b)
        result = yield depends
        return result
    return wrapper

async def get_result(func: Callable[str, str], executor: ThreadPoolExecutor, depends: Set[str]):
    return await _makeCoroutine(func, executor, depends)("hello", "world")

There are details left out about why the structure is like this, but this is the core of the problem. I am trying to type the return of the function _makeCoroutine, but as Guido said, MyPy seems to treat things one at a time, so I am guessing the decorator has not fired to update the info on the wrapper function by the time it is analyzing the return of wrapper. This leads me to the same issue above, where MyPy complains about AwaitableGenerator not matching the declared return type.

I have tried a few return types for _makeCoroutine. I can make it work if I use Callable[[str, str], Awaitable[str]], But then some of the richer info is lost, as Awaitable only documents the final return type.

[JelleZijlstra]

@types.coroutine is obsolete. I don't think we need to spend time on this.