"KeyError: 'configuration'" error in incremental mode

[jstasiak]

I'm not sure whether the name "configuration" is specific to our code (we do have a module called configuration) or is it mypy-internal thing.

CPython 3.5.1, mypy a284c48, command: python -m mypy --silent-imports --incremental --fast-parser BUNCH OF MODULES AND PACKAGES, result:

Traceback (most recent call last):
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/bin/mypy", line 6, in <module>
    main(__file__)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/main.py", line 40, in main
    res = type_check_only(sources, bin_dir, options)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/main.py", line 81, in type_check_only
    options=options)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 180, in build
    dispatch(sources, manager)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 1335, in dispatch
    process_graph(graph, manager)
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 1424, in process_graph
    stale_deps = {id for id in deps if not graph[id].is_fresh()}
  File "/var/lib/jenkins/workspace/smarkets-pylint/env3/lib/python3.5/site-packages/mypy/build.py", line 1424, in <setcomp>
    stale_deps = {id for id in deps if not graph[id].is_fresh()}
KeyError: 'configuration'

I've backed up .mypy_cache, I'll see whether I can reproduce this reliably and provide a minimal test case.

[gvanrossum]

Can you try again with the latest master from the repo, after wiping the cache? Michael has fixed a few things in this area, specifically rev c6f58b3 (the most recent HEAD).

[jstasiak]

So, two things:

• "configuration" there is a name of one of our modules
• wiping the cache will help even with the version we're using (at least temporarily), I'm trying to figure out what in the cache generates this failure

[gvanrossum]

Yeah, wiping the cache always helps for incremental-related crashes. :-)
Please do check out the very latest, Michael made another fix.

[gvanrossum]

Ping?

[jstasiak]

I'm sorry, didin't have time to reproduce/test/fiddle with it and I won't have a chance to do it in the near future, let me just close this issue and reopen in case I can repoduce it with more recent mypy.

[gvanrossum]

Thanks for keeping our tracker clean!

…

--Guido (mobile)

[gvanrossum]

I've gotten another (internal) report of an identical crash, so I'm going to reopen this.

[gvanrossum]

The internal report also gave me a full dump of the .mypy_cache from the crash.

The key thing of interest here seems to be that the module whose name appears in the KeyError message (a different one for our internal report) is not listed on the command line and hence does not get analyzed (because of --silent-imports).

Another observation: in the dump, a certain module (a sibling of the KeyError module) was suppressed. But in the crashing run, that module was added to the command line. Hm....

[gvanrossum]

OK, here's a repro.

rm -rf .mypy_cache a
mkdir a
cat >a/__init__.py <<EOF
from .b import B
from .c import C
EOF
cat >a/b.py <<EOF
class B: pass
EOF
cat >a/c.py <<EOF
class C: pass
EOF
mypy --silent-imports -i a/__init__.py
echo '#' >>a/c.py
mypy --silent-imports -i a/__init__.py a/c.py

The last line crashes with

KeyError: 'a.b'

Note that a/b.py is not checked, and it's not updated, but on the second run its name ('a.b') somehow ends up in deps.

I also have a fix, just before the offending line in build.py (stale_deps = ...) insert

deps &= set(graph)

But I am not satisfied, because I don't understand the cause of the crash, I can just prevent it.

@Michael0x2a what do you think? Here's the test case:

[case testKeyErrorCrash]
# cmd: mypy -m a
# cmd2: mypy -m a a.c
# flags: --silent-imports
[file a/__init__.py]
from .b import B
from .c import C
[file a/b.py]
class B: pass
[file a/c.py]
class C: pass
[file a/c.py.next]
class C: pass
pass

UPDATE the second: The above test case reproduces the KeyError.

[Michael0x2a]

I think I managed to isolate the error -- see #2189 for details.

I also wasn't sure why the test wasn't working, but it seems you figured it out? I can edit my pull request and copy in that test case, if you want.

Edit: I went ahead and added the test case.