Daemon anomaly

[gvanrossum]

Repro:

• check out 8cd9dbe
• python3 misc/incremental_checker.py --seed dDWZNp2XFrtdXl9vWPH8 --sample 40 --daemon -r $PWD commit 5d86e142 (When using in dmypy3 branch, add --last 40.)

Test output:

Assuming mypy is located at /Users/jukka/src/review-mypy
Temp repo will be cloned at /Users/jukka/src/review-mypy/tmp_repo
Testing file/dir located at /Users/jukka/src/review-mypy/tmp_repo/mypy
Using cache data located at /Users/jukka/src/review-mypy/.incremental_checker_cache.json

Cloning repo /Users/jukka/src/review-mypy to /Users/jukka/src/review-mypy/tmp_repo
Fetching last 40 commits (or all, if there are fewer commits than n)
Fetching commits starting at 5d86e142
Sampled down to 40 commits using random seed dDWZNp2XFrtdXl9vWPH8
...
Starting daemon
Note: first commit is evaluated twice to warm up cache
Now testing commit 9d10fcda: "Fix test failures caused by merging of #4059 (#4138)"
    Output matches expected result!
    Incremental: 12.147 sec
    Original:    13.906 sec
Now testing commit 9d10fcda: "Fix test failures caused by merging of #4059 (#4138)"
    Output matches expected result!
    Incremental: 0.347 sec
    Original:    13.906 sec
Now testing commit a1ace484: "Make stubgen return status 0 on --help (#4160)"
    Output matches expected result!
    Incremental: 12.525 sec
    Original:    14.196 sec
Now testing commit 3e49ef9b: "Allow subscripted aliases at function scope (#4000)"
    Output does not match expected result!
    Expected output (14.380 sec):



    Actual output: (11.295 sec):

        mypy/semanal_pass3.py:42: error: Name 'mypy.semanal.SemanticAnalyzerPass2' is not defined
        mypy/semanal_pass3.py:375: error: "Options" has no attribute "disallow_any_generics"


Now testing commit 880bf848: "Fix recent crashes in forward refs (#4120)"
    Output matches expected result!
    Incremental: 11.135 sec
    Original:    14.020 sec
...

(and more errors later down the run).

[gvanrossum]

Oh, something weird is going on, that file (semanal_pass3.py) is deleted in that commit (3e49ef9) compared to the previous one (a1ace48). The problem is that it is still depended on by something (due to stale metadata) and somehow that brings it back to life.

[gvanrossum]

Oh, and then somehow it finds it in the parent directory (i.e. where the mypy implementation lives rather than the repo we're type-checking).

[gvanrossum]

Heh. This bug is also present in regular incremental mode!

The specific error shown:

mypy/semanal_pass3.py:42: error: Name 'mypy.semanal.SemanticAnalyzerPass2' is not defined
mypy/semanal_pass3.py:375: error: "Options" has no attribute "disallow_any_generics"

is bogus because it found the module relative to the current directory. It's easy to take the current directory out of lib_path (https://github.com/python/mypy/blob/master/mypy/build.py#L182). With that modification I get this error:

tmp_repo/mypy/server/update.py:1: error: Cannot find module named 'mypy.semanal_pass3'
tmp_repo/mypy/server/update.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)

We are getting this because there's an indirect dependency on mypy.semanal_pass3 present in the cache metadata for mypy/server/update.py, placed there by the last run just before the run that produced the error -- in that version update.py imports build.py which imports semanal_pass3.py, and in the next run update.py is unchanged, but semanal_pass3.py is removed (we're going back in time here!) and build.py no longer imports semanal_pass3.py.

But because update.py is unchanged, its dependencies, including the indirect ones, are included in the graph, and semanal_pass3.py is amongs these.

I'm still trying to understand why indirect dependencies are written to the cache metadata -- hey @Michael0x2a do you recall why? It happens here: https://github.com/python/mypy/blob/master/mypy/build.py#L1854-L1880.

[gvanrossum]

(In the end, after I fixed the issue with indirect dependencies, the issue with pwd in lib_path was a red herring, and that was still needed.)