Segfault when working with TypeQuery

[sobolevn]

• Are you reporting a bug, or opening a feature request?

There's a bug when working with TypeQuery. It is literally unusable.

• Please insert below the code you are checking with mypy,
  or a mock-up repro if the source is private. We would appreciate
  if you try to simplify your case to a minimal repro.

Just take any type inside a plugin and pass it to TypeQuery:

from mypy.plugin import FunctionContext
from mypy.types import Type as MypyType
from mypy.types import TypeQuery
from mypy.plugin import Plugin

def analyze(ctx: FunctionContext) -> MypyType:
    print(ctx.default_return_type.accept(TypeQuery(any)))
    return ctx.default_return_type

class _ReturnsPlugin(Plugin):
    def get_function_hook(self, fullname):
        if fullname == 'ex.test':
            return analyze

def plugin(version: str):
    return _ReturnsPlugin

And provide any source code to trigger this plugin:

# ex.py
def test() -> int:
     return 1

• What is the actual behavior/output?

» PYTHONFAULTHANDLER=1 mypy ex.py
Fatal Python error: Segmentation fault

Current thread 0x0000000100b155c0 (most recent call first):
  File "/Users/sobolev/Documents/github/returns/returns/contrib/mypy/_features/decorators.py", line 18 in analyze
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/mypy/__main__.py", line 8 in console_entry
  File "/Users/sobolev/Documents/github/returns/.venv/bin/mypy", line 10 in <module>
[1]    22569 segmentation fault  PYTHONFAULTHANDLER=1 mypy ex.py

• What is the behavior/output you expect?

I would expect TypeQuery to work. Without the segfault.

• What are the versions of mypy and Python you are using?
  Do you see the same issue after installing mypy from Git master?

0.780

• What are the mypy flags you are using? (For example --strict-optional)

Full list: https://github.com/dry-python/returns/blob/master/setup.cfg#L122

[emmatyping]

Mypy sets a higher recursion limit, which may be triggering this (if there is more recursion than the system stack depth) then mypy can segfault.

Can you try setting a higher stack limit?

[sobolevn]

I have tried to add

import sys
sys.setrecursionlimit(15000)

into the plugin code. Nothing changed.

[emmatyping]

@sobolevn if you are doing something that triggers infinite recursion that probably won't help. You need to set the ulimit (assuming you are on Linux) for the stack higher.

Also your example plugin is non-functional (you don't define any). If you could fix that I can try to reproduce the issue.

[sobolevn]

@ethanhs Thanks a lot for the help!

I have updated my plugin definition. But, you could probably reproduce this with any 3rd-party plugin. This happens to me in both returns and django-stubs

And about recursion size, I am not sure that this is the case, because ex,py contains just a single function with no body. I don't think that it should behave this way on this input.

[JelleZijlstra]

There doesn't seem to be any reason from the traceback to think this is related to the recursion limit. More likely it's a mypyc bug.

[emmatyping]

@JelleZijlstra at this point I'd tend to agree, but I've noticed that tracebacks don't always show everything going wrong, and I know we've seen segfaults from stack overflows before.

Sounds like @msullivan would be the best to debug this further.

[msullivan]

Yeah I'll try to take a look.

[sobolevn]

The same happens with TypeVisitor.

Here's the code that might help:

            from mypy.erasetype import erase_type
            from mypy.types import TypeVisitor, AnyType, TypeOfAny, get_proper_type

            class KindSolver(TypeVisitor):
                def visit_type_var(self, t):
                    return AnyType(TypeOfAny.special_form)

            def test(ctx: FunctionContext) -> MypyType:
                print(erase_type(ctx.default_return_type))  # existing mypy code works fine
                print(ctx.default_return_type.accept(KindSolver()))  # custom visitors fail with segfault
                return ctx.default_return_type

Output:

def (*Any, **Any) -> Any
Fatal Python error: Segmentation fault

Current thread 0x00000001187705c0 (most recent call first):
  File "/Users/sobolev/Documents/github/returns/returns/contrib/mypy/returns_plugin.py", line 145 in test
  File "/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/mypy/__main__.py", line 8 in console_entry
  File "/Users/sobolev/Documents/github/returns/.venv/bin/mypy", line 10 in <module>
[1]    48867 segmentation fault  PYTHONFAULTHANDLER=1 mypy ex.py

Basically, all .accept calls fail with the segfault.

[sobolevn]

The reverse operation visit_* works fine:

print(KindSolver().visit_any(AnyType(TypeOfAny.special_form)))

Output:

Any

---

But, when there's a call to another .accept it also fails:

print(KindSolver().visit_callable_type(ctx.default_return_type))  # segfault :(

[sobolevn]

One more thing: Type class works. It is different, because it does not contain any visitor.visit_ calls: https://github.com/python/mypy/blob/master/mypy/types.py#L139

x = Type()
print(x.accept(TypeVisitor()))

Output:

...
  File "mypy/types.py", line 139, in accept
RuntimeError: Not implemented

[sobolevn]

Any ideas where to go next? I would be happy to help. Because this issue blocks the release of Higher Kinded Types I made.

Sneak peak:

@kinded
def map_functor(
    f: Kind[K, T],
    function: Callable[[T], V],
) -> Kind[K, V]:
    return f.map(function)

reveal_type(map_functor(Maybe(1), str))

Outputs: ex.py:53: note: Revealed type is 'ex.Kind[ex.Maybe[Any], builtins.str*]'

The only thing left is to translate ex.Kind[ex.Maybe[Any], builtins.str*] into ex.Maybe[builtins.str*], but I need TypeVisitor for it 😞

[JukkaL]

It would be helpful if the crash can be narrowed down to a smaller example that can be compiled separately with mypyc. I.e., we'd need a simplified version of TypeVisitor that doesn't depend on mypy code at all, and that can be used to reproduce the crash.

[sobolevn]

I have tried different setups, nothing helps to reproduce it.

[sobolevn]

Ok, here's something. When a class marked as @trait inherits from another @trait - segfault happens.

I am not sure if this is related or not:

# bug.py
from typing import Generic, TypeVar, Callable
from abc import abstractmethod
from mypy_extensions import trait

T1 = TypeVar('T1')
T2 = TypeVar('T2')


@trait
class Visitor(Generic[T1]):
    @abstractmethod
    def visit_node(self, t: 'Node') -> T1:
        ...


@trait
class NodeVisitor(Visitor[T1]):
    def __init__(self, function: Callable[['Node'], T1]) -> None:
        self.f = function

    def visit_node(self, t: 'Node') -> T1:
        return self.f(t)


class Node(object):
    __slots__ = ('name',)

    def __init__(self, name: str) -> None:
        self.name = name

    def accept(self, visitor: 'NodeVisitor[T2]') -> 'T2':
        raise RuntimeError('Not implemented')


class DivNode(Node):
    def accept(self, visitor: 'NodeVisitor[T2]') -> 'T2':
        return visitor.visit_node(self)


def is_node(node: Node) -> bool:
    return bool(node.name)

print(DivNode('div').accept(NodeVisitor(is_node)))

Output:

» PYTHONFAULTHANDLER=1 mypyc bug.py && python -c 'import bug'
running build_ext
building 'bug' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include -I/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/mypyc/lib-rt -I/Users/sobolev/Documents/github/returns/.venv/include -I/Users/sobolev/.pyenv/versions/3.7.7/include/python3.7m -c build/__native.c -o build/temp.macosx-10.14-x86_64-3.7/build/__native.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option
clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/sobolev/.pyenv/versions/3.7.7/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/sobolev/.pyenv/versions/3.7.7/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include build/temp.macosx-10.14-x86_64-3.7/build/__native.o -o /Users/sobolev/Documents/github/returns/bug.cpython-37m-darwin.so
[1]    13343 segmentation fault  python -c 'import bug'

When the second @trait is removed - it all works.

» PYTHONFAULTHANDLER=1 mypyc bug.py && python -c 'import bug'
running build_ext
building 'bug' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include -I/Users/sobolev/Documents/github/returns/.venv/lib/python3.7/site-packages/mypyc/lib-rt -I/Users/sobolev/Documents/github/returns/.venv/include -I/Users/sobolev/.pyenv/versions/3.7.7/include/python3.7m -c build/__native.c -o build/temp.macosx-10.14-x86_64-3.7/build/__native.o -O3 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option
clang -bundle -undefined dynamic_lookup -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/sobolev/.pyenv/versions/3.7.7/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/opt/readline/lib -L/Users/sobolev/.pyenv/versions/3.7.7/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include build/temp.macosx-10.14-x86_64-3.7/build/__native.o -o /Users/sobolev/Documents/github/returns/bug.cpython-37m-darwin.so
True

[msullivan]

OK there are a handful of issues with traits at play, all involving things that I didn't expect to work but did think that we produced error messages at runtime for?

1. Instantiating traits directly doesn't work.
2. Subclassing traits from interpreted code doesn't work.

We definitely definitely need to be able to produce error messages here.
We probably want to be able to support at lease subclassing traits from interpreted code (with @mypyc_attr(allow_interpreted_subclasses=True), at least)

[msullivan]

OK, the issue here is mypyc/mypyc#655, I think

Only sort of, I guess.

[msullivan]

There are two issues here: that we don't support what this code does, and that it segfaults. I've opened mypyc/mypyc#754 for the former and submitted #9248 for the latter.

I guess three issues: that you can't use TypeQuery usefully. There's any easy fix to that, though, which is that TypeQuery doesn't need to be a trait at all. I'll fix that too.

[sobolevn]

@msullivan I have tried your fix. There's no segfault right now. Thanks!

Unfortunately, it still does not solve my problem. Because, I still cannot use TypeQuery or TypeTranslator in my own plugin.

Here's what happens right now:

from mypy.types import TypeTranslator

class KindTranslator(TypeTranslator):
    ...

Outputs:

» mypy ex.py
ex.py:8: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.790
Traceback (most recent call last):
  File "mypy/checkexpr.py", line 3766, in accept
  File "mypy/checkexpr.py", line 263, in visit_call_expr
  File "mypy/checkexpr.py", line 340, in visit_call_expr_inner
  File "mypy/checkexpr.py", line 817, in check_call_expr_with_callee_type
  File "mypy/checkexpr.py", line 895, in check_call
  File "mypy/checkexpr.py", line 876, in check_call
  File "mypy/checkexpr.py", line 988, in check_callable_call
  File "mypy/checkexpr.py", line 725, in apply_function_plugin
  File "/Users/sobolev/Documents/github/returns/returns/contrib/mypy/_features/kind.py", line 136, in kinded_call
    print(ctx.default_return_type.accept(KindTranslator()))
TypeError: interpreted classes cannot inherit from compiled
ex.py:8: : note: use --pdb to drop into pdb

Is it possible to make TypeTranlator:

• Extendable from interpreted code
• Memory safe, so that segfault won't happen

I am going to try this out by adding @trait decorator back.

---

Context on why I do need this: dry-python/returns#631

This is a very powerful tool, but plugin users a limited not to use it, because of the current API.
Some complex type manipulations are not possible (or possible with a custom rewrite of the same API).

So, I guess this should be available as a public API.

[sobolevn]

One more thing:

from mypy.type_visitor import TypeVisitor


class KindTranslator(TypeVisitor):
    ...

This also fails with TypeError: interpreted classes cannot inherit from compiled traits
But, TypeVisitor should be extendable, because it is marked as @trait. Here: https://github.com/python/mypy/blob/v0.790/mypy/type_visitor.py#L30

This looks like a bug, doesn't it?

---

Update: no, this is not a bug. @mypyc_attr(allow_interpreted_subclasses=True) is used to allow subclasses.

[Pleket]

Hi @sobolevn , I stumbled upon this issue through the documentation in your returns project. Given that it is still referenced, I am assuming that the issue still persists, making the conversion of an emulated higher kinded type to a concrete MyPy type impossible. However, it has led to several new commits regarding TypeQuery, and the idea of a TypeTranslator class. So I was wondering, what is the current status after 4 years?