Crash on latest Django Project with dmypy

[Thorbenl]

Bug Report

Canot run mypy on latest django, mypy and django-stubs version. started crashin after updating django from 3.2.8 to 3.2.9

Expected Behavior

normal typecheck run

Actual Behavior
crashes with:

Daemon started
Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 221, in serve
  File "mypy/dmypy_server.py", line 264, in run_command
  File "mypy/dmypy_server.py", line 323, in cmd_run
  File "mypy/dmypy_server.py", line 380, in check
  File "mypy/dmypy_server.py", line 410, in initialize_fine_grained
  File "mypy/build.py", line 179, in build
  File "mypy/build.py", line 253, in _build
  File "mypy/build.py", line 2638, in dispatch
  File "mypy/build.py", line 2962, in process_graph
  File "mypy/build.py", line 3071, in process_stale_scc
  File "mypy/build.py", line 2172, in finish_passes
  File "mypy/build.py", line 2220, in update_fine_grained_deps
  File "mypy/build.py", line 2211, in compute_fine_grained_deps
  File "mypy/server/deps.py", line 114, in get_dependencies
  File "mypy/nodes.py", line 305, in accept
  File "mypy/server/deps.py", line 176, in visit_mypy_file
  File "mypy/traverser.py", line 35, in visit_mypy_file
  File "mypy/nodes.py", line 950, in accept
  File "mypy/server/deps.py", line 239, in visit_class_def
  File "mypy/traverser.py", line 71, in visit_class_def
  File "mypy/nodes.py", line 1015, in accept
  File "mypy/server/deps.py", line 351, in visit_block
  File "mypy/traverser.py", line 39, in visit_block
  File "mypy/nodes.py", line 687, in accept
  File "mypy/server/deps.py", line 201, in visit_func_def
  File "mypy/traverser.py", line 54, in visit_func_def
  File "mypy/traverser.py", line 51, in visit_func
  File "mypy/nodes.py", line 1015, in accept
  File "mypy/server/deps.py", line 351, in visit_block
  File "mypy/traverser.py", line 39, in visit_block
  File "mypy/nodes.py", line 1073, in accept
  File "mypy/server/deps.py", line 403, in visit_assignment_stmt
  File "mypy/server/deps.py", line 463, in process_lvalue
  File "mypy/server/deps.py", line 495, in get_non_partial_lvalue_type
AssertionError

(Write what happened.)

Your Environment

• mypy==0.812
• django-stubs==1.8.0
• Python version used: 3.9.8
• Operating system and version: MacOS 11

[sobolevn]

Please, report it in our repo: https://github.com/typeddjango/django-stubs/

Also:

1. Minimal repro will help a lot!
2. Try 1.9.0, it might help!

Thanks!

[Thorbenl]

@sobolevn I will try. Sadly, could only reproduce it on a fresh repo ones, and then it went away. For our project tho, it never goes away, does not matter which version combination i would take with stubs and mypy :( Ima try to pick it out of our giant repo.

[christianbundy]

Having the same problem here, it started with a Pipfile.lock update recently and only affects dmypy. We're running django-stubs@1.9.0.

[christianbundy]

@Thorbenl Any chance you have aiohttp as a dependency? I dropped into pdb and here's what I saw:

(Pdb) lvalue.node.info
<TypeInfo aiohttp.http_parser.HttpParser>

After bisecting the project history, it looks like aio-libs/aiohttp@a6c7f15 as the problematic commit.

You can repro by creating mypy.ini in aiohttp with the contents:

[mypy]
cache_fine_grained = True
ignore_errors = True

And running mypy ..

(Technically you don't have to run errors, but this lets you bisect automatically.)

cc: @asvetlov

[christianbundy]

Found the problem, although I'm not completely sure why it's a problem:

- class HttpParser(abc.ABC):
+ class HttpParser(abc.ABC, Generic[_MsgT]):

[christianbundy]

Full traceback:

aiohttp/http_parser.py: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.readthedocs.io/en/stable/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.920+dev.43eecdb0969c8cb8777fa802ba53e11075b4c802
Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 8, in <module>
    sys.exit(console_entry())
  File "/usr/local/lib/python3.9/site-packages/mypy/__main__.py", line 11, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/usr/local/lib/python3.9/site-packages/mypy/main.py", line 92, in main
    res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
  File "/usr/local/lib/python3.9/site-packages/mypy/main.py", line 170, in run_build
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 180, in build
    result = _build(
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 256, in _build
    graph = dispatch(sources, manager, stdout)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2712, in dispatch
    process_graph(graph, manager)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 3043, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 3152, in process_stale_scc
    graph[id].finish_passes()
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2225, in finish_passes
    free_tree(self.tree)
  File "/usr/local/Cellar/python@3.9/3.9.8/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 1964, in wrap_context
    yield
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2222, in finish_passes
    self.update_fine_grained_deps(self.manager.fg_deps)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2272, in update_fine_grained_deps
    merge_dependencies(self.compute_fine_grained_deps(), deps)
  File "/usr/local/lib/python3.9/site-packages/mypy/build.py", line 2263, in compute_fine_grained_deps
    return get_dependencies(target=self.tree,
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 117, in get_dependencies
    target.accept(visitor)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 326, in accept
    return visitor.visit_mypy_file(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 177, in visit_mypy_file
    super().visit_mypy_file(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 37, in visit_mypy_file
    d.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1000, in accept
    return visitor.visit_class_def(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 238, in visit_class_def
    super().visit_class_def(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 73, in visit_class_def
    o.defs.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1069, in accept
    return visitor.visit_block(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 348, in visit_block
    super().visit_block(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 41, in visit_block
    s.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 727, in accept
    return visitor.visit_func_def(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 201, in visit_func_def
    super().visit_func_def(ex)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 56, in visit_func_def
    self.visit_func(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 53, in visit_func
    o.body.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1069, in accept
    return visitor.visit_block(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 348, in visit_block
    super().visit_block(o)
  File "/usr/local/lib/python3.9/site-packages/mypy/traverser.py", line 41, in visit_block
    s.accept(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/nodes.py", line 1135, in accept
    return visitor.visit_assignment_stmt(self)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 400, in visit_assignment_stmt
    self.process_lvalue(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 460, in process_lvalue
    lvalue_type = self.get_non_partial_lvalue_type(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 492, in get_non_partial_lvalue_type
    assert not lvalue.is_new_def
AssertionError: 

[christianbundy]

Minimal repro!

from typing import Generic, TypeVar

T = TypeVar('T', int, str)

class ExampleClass(Generic[T]):
    def __init__(
        self
    ) -> None:
        self.example_attribute = None

This seems to happen when:

• Fine-grained cache is enabled (e.g. manually in config, or mandatory if using dmypy)
• Some TypeVar (T) has 2+ types.
• Some class subclasses Generic[T]
• The init method on that class sets an attribute to None.

🤯

[sobolevn]

Btw, this works fine:

from typing import Generic, TypeVar

T = TypeVar('T', int, str)

class ExampleClass(Generic[T]):
    def __init__(
        self
    ) -> None:
        ...

The problem reveals itself only if self.example_attribute = None is in place.

Great insight, thanks!

[christianbundy]

Any ideas on how we might solve it? I haven't contributed to mypy, and honestly I'm not completely sure what purpose this assert has in the first place.

[sobolevn]

It happens here:

 File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 400, in visit_assignment_stmt
    self.process_lvalue(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 460, in process_lvalue
    lvalue_type = self.get_non_partial_lvalue_type(lvalue)
  File "/usr/local/lib/python3.9/site-packages/mypy/server/deps.py", line 492, in get_non_partial_lvalue_type
    assert not lvalue.is_new_def

I've never touched this part of mypy before. And these lines of code are quite old (4 years):

mypy/mypy/server/deps.py

Lines 487 to 493 in 43eecdb

	# Probably a secondary, non-definition assignment that doesn't
	# result in a non-partial type. We won't be able to infer any
	# dependencies from this so just return something. (The first,
	# definition assignment with a partial type is handled
	# differently, in the semantic analyzer.)
	assert not lvalue.is_new_def
	return UninhabitedType()

Git Blame Link

This can either be:

1. A trivial if bug, where we just forgot to add some condition
2. Quite complex one, due to our way-overly-compicated-plugin

[sobolevn]

@christianbundy this seems to remove the crash:

            if isinstance(lvalue.node, Var):
                if lvalue.node.type:
                    lvalue_type = get_proper_type(lvalue.node.type)
                else:
                    lvalue_type = NoneType()

But, I am not sure it is correct though. Do you want to send a PR? We can work from there!