Segfault when working with TypeQuery

[sobolevn]

• Are you reporting a bug, or opening a feature request?

There's a bug when working with TypeQuery. It is literally unusable.

• Please insert below the code you are checking with mypy,
  or a mock-up repro if the source is private. We would appreciate
  if you try to simplify your case to a minimal repro.

Just take any type inside a plugin and pass it to TypeQuery:

from mypy.plugin import FunctionContext
from mypy.types import Type as MypyType
from mypy.types import TypeQuery
from mypy.plugin import Plugin

def analyze(ctx: FunctionContext) -> MypyType:
    print(ctx.default_return_type.accept(TypeQuery(any)))
    return ctx.default_return_type

class _ReturnsPlugin(Plugin):
    def get_function_hook(self, fullname):
        if fullname == 'ex.test':
            return analyze

def plugin(version: str):
    return _ReturnsPlugin

And provide any source code to trigger this plugin:

# ex.py
def test() -> int:
     return 1

• What is the actual behavior/output?

» PYTHONFAULTHANDLER=1 mypy ex.py
Fatal Python error: Segmentation fault

Current thread 0x0000000100b155c0 (most recent call first):
  File "/Users/sobolev/Documents/github/returns/returns/contrib/mypy/_features/decorators.py", line 18 in analyze
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/mypy/__main__.py", line 8 in console_entry
  File "/Users/sobolev/Documents/github/returns/.venv/bin/mypy", line 10 in <module>
[1]    22569 segmentation fault  PYTHONFAULTHANDLER=1 mypy ex.py

• What is the behavior/output you expect?

I would expect TypeQuery to work. Without the segfault.

• What are the versions of mypy and Python you are using?
  Do you see the same issue after installing mypy from Git master?

0.780

• What are the mypy flags you are using? (For example --strict-optional)

Full list: https://github.com/dry-python/returns/blob/master/setup.cfg#L122

[emmatyping]

Mypy sets a higher recursion limit, which may be triggering this (if there is more recursion than the system stack depth) then mypy can segfault.

Can you try setting a higher stack limit?

[sobolevn]

I have tried to add

import sys
sys.setrecursionlimit(15000)

into the plugin code. Nothing changed.

[emmatyping]

@sobolevn if you are doing something that triggers infinite recursion that probably won't help. You need to set the ulimit (assuming you are on Linux) for the stack higher.

Also your example plugin is non-functional (you don't define any). If you could fix that I can try to reproduce the issue.

[sobolevn]

@ethanhs Thanks a lot for the help!

I have updated my plugin definition. But, you could probably reproduce this with any 3rd-party plugin. This happens to me in both returns and django-stubs

And about recursion size, I am not sure that this is the case, because ex,py contains just a single function with no body. I don't think that it should behave this way on this input.

[JelleZijlstra]

There doesn't seem to be any reason from the traceback to think this is related to the recursion limit. More likely it's a mypyc bug.

[emmatyping]

@JelleZijlstra at this point I'd tend to agree, but I've noticed that tracebacks don't always show everything going wrong, and I know we've seen segfaults from stack overflows before.

Sounds like @msullivan would be the best to debug this further.

[msullivan]

Yeah I'll try to take a look.

[sobolevn]

The same happens with TypeVisitor.

Here's the code that might help:

            from mypy.erasetype import erase_type
            from mypy.types import TypeVisitor, AnyType, TypeOfAny, get_proper_type

            class KindSolver(TypeVisitor):
                def visit_type_var(self, t):
                    return AnyType(TypeOfAny.special_form)

            def test(ctx: FunctionContext) -> MypyType:
                print(erase_type(ctx.default_return_type))  # existing mypy code works fine
                print(ctx.default_return_type.accept(KindSolver()))  # custom visitors fail with segfault
                return ctx.default_return_type

Output:

def (*Any, **Any) -> Any
Fatal Python error: Segmentation fault

Current thread 0x00000001187705c0 (most recent call first):
  File "/Users/sobolev/Documents/github/returns/returns/contrib/mypy/returns_plugin.py", line 145 in test
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/mypy/__main__.py", line 8 in console_entry
  File "/Users/sobolev/Documents/github/returns/.venv/bin/mypy", line 10 in <module>
[1]    48867 segmentation fault  PYTHONFAULTHANDLER=1 mypy ex.py

Basically, all .accept calls fail with the segfault.

[sobolevn]

The reverse operation visit_* works fine:

print(KindSolver().visit_any(AnyType(TypeOfAny.special_form)))

Output:

Any

---

But, when there's a call to another .accept it also fails:

print(KindSolver().visit_callable_type(ctx.default_return_type))  # segfault :(

[sobolevn]

One more thing: Type class works. It is different, because it does not contain any visitor.visit_ calls: https://github.com/python/mypy/blob/master/mypy/types.py#L139

x = Type()
print(x.accept(TypeVisitor()))

Output:

...
  File "mypy/types.py", line 139, in accept
RuntimeError: Not implemented

[sobolevn]

Any ideas where to go next? I would be happy to help. Because this issue blocks the release of Higher Kinded Types I made.

Sneak peak:

@kinded
def map_functor(
    f: Kind[K, T],
    function: Callable[[T], V],
) -> Kind[K, V]:
    return f.map(function)

reveal_type(map_functor(Maybe(1), str))

Outputs: ex.py:53: note: Revealed type is 'ex.Kind[ex.Maybe[Any], builtins.str*]'

The only thing left is to translate ex.Kind[ex.Maybe[Any], builtins.str*] into ex.Maybe[builtins.str*], but I need TypeVisitor for it 😞

[JukkaL]

It would be helpful if the crash can be narrowed down to a smaller example that can be compiled separately with mypyc. I.e., we'd need a simplified version of TypeVisitor that doesn't depend on mypy code at all, and that can be used to reproduce the crash.

[sobolevn]

I have tried different setups, nothing helps to reproduce it.