New incremental mode (-i) crasher

[gvanrossum]

I've seen this a few times myself and now it's being reported. The traceback ends with

  File "/Users/x/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/types.py", line 388, in accept
    return visitor.visit_instance(self)
  File "/Users/x/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/fixup.py", line 153, in visit_instance
    node = lookup_qualified(self.modules, type_ref, self.quick_and_dirty)
  File "/Users/x/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/fixup.py", line 237, in lookup_qualified
    stnode = lookup_qualified_stnode(modules, name, quick_and_dirty)
  File "/Users/x/src/server/.mypy/venv/lib/python3.6/site-packages/mypy/fixup.py", line 252, in lookup_qualified_stnode
    assert '.' in head, "Cannot find %s" % (name,)
AssertionError: Cannot find _importlib_modulespec.ModuleType

[ilevkivskyi]

Probably this is a "leftover" that is still not fixed by #3203
@pkch Could you please take a look?

[gvanrossum]

Oh, I'll also check if the user who reported this had a sufficiently recent mypy version.

[gvanrossum]

The user who reported this was running an older version of mypy, so there's nothing to see and nothing to do here.

[UPDATE: I was wrong there]

[pkch]

Unfortunately, this is the recent version (after my fix). I can see from the line numbers. Is there any information about the crash (command line, whether it's intermittent, warm/cold cache, etc)?

[gvanrossum]

Well, actually the user reported they got the crash after upgrading to mypy 1d1c62d (and they have the Terminal window logs to prove it to me :-) so I'm afraid I need to reopen this. Sorry for the quick cycling.

[gvanrossum]

The user also reported: It [mypy -i] worked the first two times, then he added # type: ignore to the (previously empty) __init__.py of a very commonly used toplevel package, then it crashed.

[gvanrossum]

Maybe also important, we have a script that computes the list of files to pass to mypy, and the offending __init__.py file (let's call it dropbox/__init__.py) was previously empty, so the result of adding # type: ignore to it was to include it in the command line. Many submodules of dropbox do have type annotations and are included in both runs.

Also, we run with --follow-imports=skip --ignore-missing-imports . This implies that the dropbox.__init__.py file was not analyzed in any way in the runs before the crash, but it was analyzed in the crashing run.

[gvanrossum]

Also important: Python 2 mode!

[ilevkivskyi]

Also important: Python 2 mode!

This is the key I think.

[ilevkivskyi]

@pkch maybe the solution is just to skip the reverse renaming in the PY2 mode?

[pkch]

Yup, python 2 doesn't use _importlib_modulespec at all.

[gvanrossum]

Do you have time to make a fix?

[pkch]

Yes. I can't reproduce the crash, but I'm pretty sure what @ilevkivskyi said is sufficient. If it still crashes, we'll just have to look further.

[gvanrossum]

Please send a PR -- I have a repro but I can't publish it, so I'll test your PR against it once I have it.

[pkch]

I used the simplest fix I could (just check that _importlib_modulespec exists before redirecting). But I guess it's fine, since this is a very low-level implementation detail.