Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 31 additions & 11 deletions mypy/checker.py
Original file line number Diff line number Diff line change
Expand Up @@ -430,6 +430,7 @@ def visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None:

def _visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None:
num_abstract = 0
num_awaitable_coroutine = 0
if not defn.items:
# In this case we have already complained about none of these being
# valid overloads.
Expand All @@ -445,8 +446,22 @@ def _visit_overloaded_func_def(self, defn: OverloadedFuncDef) -> None:
self.check_func_item(fdef.func, name=fdef.func.name())
if fdef.func.is_abstract:
num_abstract += 1
if fdef.func.is_awaitable_coroutine:
num_awaitable_coroutine += 1
if num_abstract not in (0, len(defn.items)):
self.fail(message_registry.INCONSISTENT_ABSTRACT_OVERLOAD, defn)
if num_awaitable_coroutine not in (0, len(defn.items)):
self.fail(message_registry.INCONSISTENT_COROUTINE_OVERLOAD, defn)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test that goes through this branch?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a basic test:

Details
[case testOverloadedCoroutineVariantMissingDecorator1]
from foo import *
[file foo.pyi]
from types import coroutine
from typing import overload

@coroutine \
# E: Overloaded method has both coroutine and non-coroutine variants
@overload
def f(x: int) -> int: pass
@overload
def f(x: str) -> str: pass
[out]

and I got a massive traceback:

Details
__________________________________________________________________ testOverloadedCoroutineVariantMissingDecorator1 __________________________________________________________________
[gw4] darwin -- Python 3.7.3 /Users/myk/.virtualenvs/mypy/bin/python3.7
data: /Users/myk/dev/mypy/test-data/unit/check-overloading.test:5046:
SystemExit: 2
------------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------------
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 8, in <module>
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/execnet/gateway_base.py", line 1525, in serve
    SlaveGateway(io=io, id=id, _startcount=2).serve()
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/execnet/gateway_base.py", line 1038, in serve
    self._execpool.integrate_as_primary_thread()
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/execnet/gateway_base.py", line 250, in integrate_as_primary_thread
    self._perform_spawn(reply)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/execnet/gateway_base.py", line 268, in _perform_spawn
    reply.run()
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/execnet/gateway_base.py", line 204, in run
    self._result = func(*args, **kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/execnet/gateway_base.py", line 1063, in executetask
    do_exec(co, loc)  # noqa
  File "<string>", line 1, in do_exec
  File "<remote exec>", line 249, in <module>
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/main.py", line 242, in pytest_cmdline_main
    return wrap_session(config, _main)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/main.py", line 209, in wrap_session
    session.exitstatus = doit(config, session) or 0
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/main.py", line 249, in _main
    config.hook.pytest_runtestloop(session=session)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "<remote exec>", line 67, in pytest_runtestloop
  File "<remote exec>", line 84, in run_one_test
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 78, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 93, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 173, in call_and_report
    call = call_runtest_hook(item, when, **kwds)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 198, in call_runtest_hook
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 226, in from_call
    result = func()
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 198, in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 68, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/manager.py", line 62, in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/pluggy/callers.py", line 187, in _multicall
    res = hook_impl.function(*args)
  File "/Users/myk/.virtualenvs/mypy/lib/python3.7/site-packages/_pytest/runner.py", line 123, in pytest_runtest_call
    item.runtest()
  File "/Users/myk/dev/mypy/mypy/test/data.py", line 223, in runtest
    suite.run_case(self)
  File "/Users/myk/dev/mypy/mypy/test/testcheck.py", line 124, in run_case
    self.run_case_once(testcase)
  File "/Users/myk/dev/mypy/mypy/test/testcheck.py", line 186, in run_case_once
    alt_lib_path=test_temp_dir)
  File "/Users/myk/dev/mypy/mypy/build.py", line 166, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr)
  File "/Users/myk/dev/mypy/mypy/build.py", line 226, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/myk/dev/mypy/mypy/build.py", line 2572, in dispatch
    process_graph(graph, manager)
  File "/Users/myk/dev/mypy/mypy/build.py", line 2885, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/myk/dev/mypy/mypy/build.py", line 2992, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/myk/dev/mypy/mypy/build.py", line 2101, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/myk/dev/mypy/mypy/checker.py", line 285, in check_first_pass
    self.accept(d)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 396, in accept
    stmt.accept(self)
  File "/Users/myk/dev/mypy/mypy/nodes.py", line 497, in accept
    return visitor.visit_overloaded_func_def(self)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 429, in visit_overloaded_func_def
    self._visit_overloaded_func_def(defn)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 446, in _visit_overloaded_func_def
    self.check_func_item(fdef.func, name=fdef.func.name())
  File "/Users/myk/dev/mypy/mypy/checker.py", line 788, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 871, in check_func_def
    typ = self.get_awaitable_coroutine_return_type(defn, typ)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 983, in get_awaitable_coroutine_return_type
    [ty, tc, tr, t])
  File "/Users/myk/dev/mypy/mypy/checker.py", line 3669, in named_generic_type
    info = self.lookup_typeinfo(name)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 3675, in lookup_typeinfo
    sym = self.lookup_qualified(fullname)
  File "/Users/myk/dev/mypy/mypy/checker.py", line 3737, in lookup_qualified
    raise KeyError(msg.format(last, name))
KeyError: "Failed qualified lookup: 'AwaitableGenerator' (fullname = 'typing.AwaitableGenerator')."
------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------
tmp/foo.pyi:4: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.710+dev.1980f6a8af1f1fd556e42dcb7cc108071b7167ac.dirty
tmp/foo.pyi:4: : note: use --pdb to drop into pdb

More work is definitely needed here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of this might just be an issue with the text fixtures (tests don't always get run with full typing). I thought we showed a special error message in that case though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought we showed a special error message in that case though.

We have custom nice error messages only for basic things like list or tuple missing in the fixture. I think this purely a fixture problem.

# If items contains coroutines and check_func_item fixed their type,
# also fix the overload type.
if num_awaitable_coroutine:
assert isinstance(defn.type, Overloaded)
types = []
for fdef, typ in zip(defn.items, defn.type.items()):
assert isinstance(fdef, Decorator)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we could get a crash here if one of the overloads doesn't have @coroutine.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah -- maybe we could just skip this entire block if @coroutine is applied inconsistently? It's probably not worth trying to infer more precise types once we know the decorators are wonky.

types.append(
self.get_awaitable_coroutine_return_type(fdef.func, typ))
defn.type = Overloaded(types)
if defn.impl:
defn.impl.accept(self)
if defn.info:
Expand Down Expand Up @@ -853,17 +868,7 @@ def check_func_def(self, defn: FuncItem, typ: CallableType, name: Optional[str])
if defn.is_awaitable_coroutine:
# Update the return type to AwaitableGenerator.
# (This doesn't exist in typing.py, only in typing.pyi.)
t = typ.ret_type
c = defn.is_coroutine
ty = self.get_generator_yield_type(t, c)
tc = self.get_generator_receive_type(t, c)
if c:
tr = self.get_coroutine_return_type(t)
else:
tr = self.get_generator_return_type(t, c)
ret_type = self.named_generic_type('typing.AwaitableGenerator',
[ty, tc, tr, t])
typ = typ.copy_modified(ret_type=ret_type)
typ = self.get_awaitable_coroutine_return_type(defn, typ)
defn.type = typ

# Push return type.
Expand Down Expand Up @@ -963,6 +968,21 @@ def check_func_def(self, defn: FuncItem, typ: CallableType, name: Optional[str])

self.binder = old_binder

def get_awaitable_coroutine_return_type(self,
defn: FuncItem,
typ: CallableType) -> CallableType:
t = typ.ret_type
c = defn.is_coroutine
ty = self.get_generator_yield_type(t, c)
tc = self.get_generator_receive_type(t, c)
if c:
tr = self.get_coroutine_return_type(t)
else:
tr = self.get_generator_return_type(t, c)
ret_type = self.named_generic_type('typing.AwaitableGenerator',
[ty, tc, tr, t])
return typ.copy_modified(ret_type=ret_type)

def is_forward_op_method(self, method_name: str) -> bool:
if self.options.python_version[0] == 2 and method_name == '__div__':
return True
Expand Down
2 changes: 2 additions & 0 deletions mypy/message_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@
CANNOT_ASSIGN_TO_TYPE = 'Cannot assign to a type' # type: Final
INCONSISTENT_ABSTRACT_OVERLOAD = \
'Overloaded method has both abstract and non-abstract variants' # type: Final
INCONSISTENT_COROUTINE_OVERLOAD = \
'Overloaded method has both coroutine and non-coroutine variants' # type: Final
MULTIPLE_OVERLOADS_REQUIRED = 'Single overload definition, multiple required' # type: Final
READ_ONLY_PROPERTY_OVERRIDES_READ_WRITE = \
'Read-only property cannot override read-write property' # type: Final
Expand Down
15 changes: 15 additions & 0 deletions test-data/unit/check-overloading.test
Original file line number Diff line number Diff line change
Expand Up @@ -5001,3 +5001,18 @@ def f(x):

reveal_type(f(g([]))) # E: Revealed type is 'builtins.list[builtins.int]'
[builtins fixtures/list.pyi]

[case testTypeCheckOverloadCoroutine]
from types import coroutine
from typing import overload
@overload
@coroutine
def f(x: int) -> None: ...
@overload
@coroutine
def f(x: str) -> None: ...
def f(x): pass
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we should maybe also include a check that enforces that @coroutine is included in the implementation as well? Or more generally, mandate that decorators used consistently across all overload signatures + the implementation.

Normally, we don't check whether the implementation signature actually matches the given overload variants because it's too difficult to do so. But it might be worth the extra effort to make sure at least decorators in general are applied consistently: it's not as hard to check, and not including the @coroutine decorator on the implementation can result in some potentially confusing errors at runtime.

The fact that we currently don't do this for @abstractmethod is an oversight, I think -- it doesn't really make a major difference at runtime whether f actually is abstract or not (especially since mypy's abstract checks largely supersede the same ones done at runtime by the abc module), so probably nobody bothered reporting this.

This is probably out-of-scope for this PR though. Up to you if you want to try tackling this or not -- but if you don't have time, could you file an issue so we can keep track of this?


reveal_type(f) # E: Revealed type is 'Overload(def (x: builtins.int) -> typing.AwaitableGenerator[Any, Any, Any, None], def (x: builtins.str) -> typing.AwaitableGenerator[Any, Any, Any, None])'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add test that show that the overload actually works as expected? For example, test that await f(1) works, but await f([]) does not.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more request for more test cases: could you try checking to see what happens when you try using @abstractmethod with @coroutine? (E.g. make sure we still get the correct types, make sure the result makes sense no matter which decorator you apply first)

[builtins fixtures/dict.pyi]
[typing fixtures/typing-full.pyi]