stubgen and C Extensions (pyd)

[kdschlosser]

I work on a project called python-openzwave. I have rewritten the build system for this library. The library makes use of a CPP library called OpenZWave. This library gets compiled by the build system. Cython is what is used for the "bridge" between python code and the cpp code in OpenZWave. The output of this process is a single pyd file. I would like to be able to create a stub file for this pyd file. I have encountered an issue when creating documentation for the library where if the library is already installed on the system and the build system imports the newly created pyd file python crashes. So a direct import of the new file cannot be done. Using the code example below I am able to import the new library and avoid the application crash.

import os
import imp

mod_file = os.path.join(build_dir, 'libopenzwave.pyd')
mod = imp.load_dynamic(
    'libopenzwave',
    mod_file
)
    
sys.modules['libopenzwave'] = mod

I am unsure of how stubgen handles the importation of c extensions. I am needing to build the stub file at build time so I have subclassed setuptools.Command to acomplish this task,

the code below is what is used in the run method of the class

        from mypy import stubgen
        import sys

        builder = self.distribution.get_command_obj('build')
        build_path = builder.build_lib
        build_ext = self.distribution.get_command_obj('build_ext')
        extension = self.distribution.ext_modules[0]
        ext_path = build_ext.get_ext_fullpath(extension.name)

        if sys.version_info[0] < 3:
            args = ['--py2']
        else:
            args = []

        args += [
            '--include-private',
            '--output',
            build_path,
            ext_path
        ]

        options = stubgen.parse_options(args)
        stubgen.generate_stubs(options)

when this runs the program exits with the following error

Critical error during semantic analysis: mypy: can't decode file 'build\lib.win-amd64-3.7\libopenzwave.pyd': 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

using the positional file is the only way to be able to direct the program to load the proper pyd file. Is there a special way to go about passing the path and filename of the c extension that I want to create the stub file for? or do I have to use the imp module as outlines in the code block above and set the module into sys.modules and then pass the module name to stubgen?

[kdschlosser]

OK so I did what I mentioned above. I placed the new extension file into sys.modules.. then passed the module name using --module. and now I am getting this traceback

Nothing to do?!
Traceback (most recent call last):
  File "setup.py", line 622, in <module>
    "License :: OSI Approved :: "
  File "\python37\lib\site-packages\setuptools\__init__.py", line 143, in setup
    return distutils.core.setup(**attrs)
  File "\python37\lib\distutils\core.py", line 148, in setup
    dist.run_commands()
  File "\python37\lib\distutils\dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "\python37\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "\python-openzwave\pyozw_build\build.py", line 74, in run
    self.run_command(sub_command)
  File "\python37\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "\python37\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "\python-openzwave\pyozw_build_ext.py", line 59, in run
    self.run_command('build_bootstrap')
  File "\python37\lib\distutils\cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "\python37\lib\distutils\dist.py", line 985, in run_command
    cmd_obj.run()
  File "\python-openzwave\pyozw_build_bootstrap.py", line 360, in run
    stubgen.generate_stubs(options)
  File "mypy\stubgen.py", line 1100, in generate_stubs
  File "mypy\stubgenc.py", line 44, in generate_stub_for_c_module
  File "mypy\stubgenc.py", line 153, in generate_c_function_stub
TypeError: str object expected; got None

[JukkaL]

-m or --module is the way to generate stubs of a C/C++ extension module. You can use --search-path or --python-executable to help it find the module.

It looks like you hit a bug though -- it shouldn't crash.

[JukkaL]

Is the project open source? Can you provide detailed instructions for reproducing the crash?

[kdschlosser]

https://github.com/kdschlosser/python-openzwave/tree/mypy-test

The build process is pretty involved.. don't run this on a raspberry pi or it will take over an hour to compile....

this is the command to run when building it. This installs nothing into the site-packages folder of the interpreter you use.

python setup.py build --dev

if you are running on a fairly new machine it should take 2-3 minutes for it to complete. The build system works for the OS's listed below.

here is the list of requirements that the build system does not handle.

• Windows: Visual Studio 2010 or newer with Windows SDK for the VS version. Visual Studio Build Tools will also work.
• cygwin: gcc, g++, ar, ld, ranlib
• darwin (OSX): clang, clang++, ar, ld, ranlib
• freebsd: cc, c++, ar, ld, ranlib
• linux (assortment of flavors): gcc, g++, ar, ld, ranlib

make is not needed everything is handled by the setup program (much faster).

From the last error it appears as tho it is finding the module properly. I didn't look at the code to see how the loading of the module is done.

[zacps]

The later error: TypeError: str object expected; got None also occurs when attempting to generate stubs for _ldap (a shim to libldap used by ldap).

[JukkaL]

Thanks for the instructions! Here's an easy way to reproduce the crash:

$ pip install python-ldap  # You may need to install some C deps first
$ stubgen --module _ldap
Traceback (most recent call last):
  File "/home/jukka/venv/bin/stubgen", line 8, in <module>
    sys.exit(main())
  File "mypy/stubgen.py", line 1559, in main
  File "mypy/stubgen.py", line 1452, in generate_stubs
  File "mypy/stubgenc.py", line 44, in generate_stub_for_c_module
  File "mypy/stubgenc.py", line 151, in generate_c_function_stub
TypeError: str object expected; got None

[jethron]

I also get a similar error with the psycopg2._psycopg module from psycopg2-binary.

Moving the:

• find_sources.cpython-37m-x86_64-linux-gnu.so
• modulefinder.cpython-37m-x86_64-linux-gnu.so
• stubdoc.cpython-37m-x86_64-linux-gnu.so
• stubgenc.cpython-37m-x86_64-linux-gnu.so
• stubgen.cpython-37m-x86_64-linux-gnu.so

files temporarily out of <virtualenv>/lib/python3.7/site-packages/mypy/ (presumably forces a fallback to the Python implementation) generates a stub for that module (not an entirely valid one, but better than a crash?). Segfaults for other, presumably non-C modules, though, so had to put them back when done.

[martok]

Ì get the same error with package av.
Moving the files mentioned by @jethron does not help here, instead I get a segfault. Stacktrace:

Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x00000000005d01e5 in _PyDict_GetItemIdWithError ()
(gdb) bt
#0  0x00000000005d01e5 in _PyDict_GetItemIdWithError ()
#1  0x00000000005ea3b7 in _PyFrame_New_NoTrack ()
#2  0x00000000004e43e6 in PyFrame_New ()
#3  0x00007ffff58c678c in CPy_AddTraceback (filename=<optimized out>, funcname=<optimized out>, line=513, globals=<optimized out>) at /io/mypy/mypyc/lib-rt/CPy.h:1138
#4  0x00007ffff590014c in CPyDef_modulefinder___compute_search_paths (cpy_r_sources=0x7fffeae33280, cpy_r_options=<optimized out>, cpy_r_data_dir=<optimized out>, cpy_r_alt_lib_path=0x903a20 <_Py_NoneStruct>) at build/__native_56fa58acf89604978a41.c:763364
#5  0x00007ffff58fc282 in CPyDef_mypy___build____build (cpy_r_sources=0x7fffeae33280, cpy_r_options=<optimized out>, cpy_r_alt_lib_path=0x903a20 <_Py_NoneStruct>, cpy_r_flush_errors=0x7fffeecd1680, cpy_r_fscache=<optimized out>, cpy_r_stdout=0x7ffff6c0de10, 
    cpy_r_stderr=0x7ffff6c0dee0, cpy_r_extra_plugins=0x7ffff2d91740) at build/__native_56fa58acf89604978a41.c:34647
#6  0x00007ffff58fbad4 in CPyDef_mypy___build___build (cpy_r_sources=<optimized out>, cpy_r_options=<optimized out>, cpy_r_alt_lib_path=<optimized out>, cpy_r_flush_errors=<optimized out>, cpy_r_fscache=<optimized out>, cpy_r_stdout=0x7ffff6c0de10, 
    cpy_r_stderr=0x7ffff6c0dee0, cpy_r_extra_plugins=0x7ffff2d91740) at build/__native_56fa58acf89604978a41.c:33852
#7  0x00007ffff58fdfb9 in CPyPy_mypy___build___build (self=<optimized out>, args=<optimized out>, kw=<optimized out>) at build/__native_56fa58acf89604978a41.c:34275
#8  0x00000000005f25e5 in PyCFunction_Call ()
#9  0x00000000005f31fe in _PyObject_MakeTpCall ()
#10 0x000000000056c086 in _PyEval_EvalFrameDefault ()
#11 0x00000000005f298b in _PyFunction_Vectorcall ()
#12 0x0000000000567058 in _PyEval_EvalFrameDefault ()
#13 0x0000000000565332 in _PyEval_EvalCodeWithName ()
#14 0x00000000005f2b85 in _PyFunction_Vectorcall ()
#15 0x0000000000567058 in _PyEval_EvalFrameDefault ()
#16 0x00000000005f298b in _PyFunction_Vectorcall ()
#17 0x0000000000567058 in _PyEval_EvalFrameDefault ()
#18 0x0000000000565332 in _PyEval_EvalCodeWithName ()
#19 0x0000000000687033 in PyEval_EvalCode ()
#20 0x0000000000677fa1 in ?? ()
#21 0x000000000067801f in ?? ()
#22 0x00000000006780d7 in PyRun_FileExFlags ()
#23 0x000000000067845a in PyRun_SimpleFileExFlags ()
#24 0x00000000006af6ce in Py_RunMain ()
#25 0x00000000006afa59 in Py_BytesMain ()
#26 0x00007ffff7dec0b3 in __libc_start_main (main=0x4ec760 <main>, argc=6, argv=0x7fffffffe148, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe138) at ../csu/libc-start.c:308
#27 0x00000000005f6ede in _start ()

Installing the interpreted version of mypy instead gives this error:

Traceback (most recent call last):
  File "/home/sh/.local/bin/stubgen", line 8, in <module>
    sys.exit(main())
  File "/home/sh/.local/lib/python3.8/site-packages/mypy/stubgen.py", line 1564, in main
    generate_stubs(options)
  File "/home/sh/.local/lib/python3.8/site-packages/mypy/stubgen.py", line 1457, in generate_stubs
    generate_stub_for_c_module(mod.module, target, sigs=sigs, class_sigs=class_sigs)
  File "/home/sh/.local/lib/python3.8/site-packages/mypy/stubgenc.py", line 51, in generate_stub_for_c_module
    generate_c_type_stub(module, name, obj, types, imports=imports, sigs=sigs,
  File "/home/sh/.local/lib/python3.8/site-packages/mypy/stubgenc.py", line 281, in generate_c_type_stub
    all_bases = obj.mro()
TypeError: descriptor 'mro' of 'type' object needs an argument