Fix accessing qualified import in incremental mode #3548
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit fixes python#3274 The problem was that mypy was previously doing the following, given an empty cache: 1. Analyze the SCCs (ignoring the builtins) in this exact order: `['c.d']`, then `['c']`, then `['b']`, then `['a']`. No issues here. 2. Parse, typecheck, and write `c.d` to cache -- also no issues here. 2. Parse, typecheck, and write `c` to cache. The error occurs here -- mypy doesn't recognize that `c` has any submodules, and so will not record `c.d` into `c`'s symbol table. This means the saved cache for `c` will be essentially empty. 3. When parsing `b`, mypy *will* recognize that `c.d` is a submodule of `c` due to the import. During the semantic analysis phase, mypy will then actually modify `c`'s symbol table to include `d`. This exact process takes place in `SemanticAnalyzer.add_submodules_to_parent_modules`. This is why typechecking succeeds for `a` and `b` during a fresh However, this change wasn't ever written to the cache, so won't be remembered in the next run! 4. Will parse and typecheck `a`, using the modified (but not preserved) symbol table. Or to put it more succinctly, the code sometimes seems to be relying on the assumption that a symbol table for a given module will not be modified after that SCC is processed. However, this invariant is false due to the 'parent patching' mechanism. It's worth nothing that this patching also occurs during Python's runtime -- it isn't just an artifact of mypy's parsing process. This commit opts for a relatively conservative course of action by simply re-running this patching process when handling fresh SCCs. Other potential fixes include deferring writing to cache until *all* SCCs are processed (which initially seemed like a more robust solution but broke multiple tests when I tried it), or replacing the current parent patching mechanism with something entirely different (which seems like the sort of thing that could subtly break code).
|
Thanks! I agree with your choice of fix. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request fixes #3274
The problem was that mypy was previously doing the following, given an empty cache:
['c.d'], then['c'], then['b'], then['a']. No issues here.c.dto cache -- also no issues here.cto cache. The error occurs here -- mypy doesn't recognize thatchas any submodules, and so will not recordc.dintoc's symbol table. This means the saved cache forcwill be essentially empty.b, mypy will recognize thatc.dis a submodule ofcdue to the import. During the semantic analysis phase (more precisely, inSemanticAnalyzer.add_submodules_to_parent_modules), mypy will actually modifyc's symbol table to included, which is why typechecking succeeds forbandaduring a fresh run. However, this change wasn't ever written to the cache, so won't be remembered when re-running incremental mode!a, using the modified (but not preserved) symbol table.Or to put it more succinctly, the code sometimes seems to be relying on the assumption that a symbol table for a given module will not be modified after that SCC is processed. However, this invariant is false due to the 'parent patching' mechanism.
This commit opts for a relatively conservative course of action by simply re-running this patching process when handling fresh SCCs.
Other potential fixes I considered included deferring writing to cache until all SCCs are processed to try and preserve this info and restore the above invariant (which initially seemed like a more robust solution but broke multiple tests when I tried it), or replacing the current parent patching mechanism with something entirely different (which seems like the sort of thing that could subtly break code).