Curious build failure with Python 3.7 on Windows

[dnicolodi]

Running python -m build . in GitHub CI ends with this error:

+ meson install --destdir C:\Users\RUNNER~1\AppData\Local\Temp\build-via-sdist-x9mu6s_a\siphash24-1.1.dev0\.mesonpy-wgvn__mt\install
ninja: Entering directory `C:\Users\RUNNER~1\AppData\Local\Temp\build-via-sdist-x9mu6s_a\siphash24-1.1.dev0\.mesonpy-wgvn__mt\build'
ninja: no work to do.
Installing siphash24.pyd to C:\Users\RUNNER~1\AppData\Local\Temp\build-via-sdist-x9mu6s_a\siphash24-1.1.dev0\.mesonpy-wgvn__mt\install\hostedtoolcache\windows\Python\3.7.9\x64\Lib\site-packages
Installing siphash24.lib to C:\Users\RUNNER~1\AppData\Local\Temp\build-via-sdist-x9mu6s_a\siphash24-1.1.dev0\.mesonpy-wgvn__mt\install\hostedtoolcache\windows\Python\3.7.9\x64\Lib\site-packages
    return mesonpy._tags.StableABITag(tag)

ERROR Backend subprocess exited when trying to invoke build_wheel
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 310, in _calculate_file_abi_tag_heuristic_windows
    return mesonpy._tags.StableABITag(tag)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\_tags.py", line 37, in __init__
    raise ValueError(f'Invalid PEP 3149 stable ABI tag, expecting pattern `{self._REGEX.pattern}`')
ValueError: Invalid PEP 3149 stable ABI tag, expecting pattern `^abi(?P<abi_number>[0-9]+)$`

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\pep517\in_process\_in_process.py", line 351, in <module>
    main()
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\pep517\in_process\_in_process.py", line 333, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "C:\hostedtoolcache\windows\Python\3.7.9\x64\lib\site-packages\pep517\in_process\_in_process.py", line 250, in build_wheel
    metadata_directory)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 987, in build_wheel
    return project.wheel(out).name
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 935, in wheel
    wheel = self._wheel_builder.build(self._build_dir)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 578, in build
    wheel_file = pathlib.Path(directory, f'{self.name}.whl')
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 189, in name
    python_tag=self.python_tag,
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 244, in python_tag
    selected_tag = self._select_abi_tag()
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 375, in _select_abi_tag
    tags = self._files_by_tag()
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 361, in _files_by_tag
    tag = self._calculate_file_abi_tag_heuristic(file)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 346, in _calculate_file_abi_tag_heuristic
    return self._calculate_file_abi_tag_heuristic_windows(filename)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\__init__.py", line 312, in _calculate_file_abi_tag_heuristic_windows
    return mesonpy._tags.InterpreterTag(tag)
  File "C:\Users\RUNNER~1\AppData\Local\Temp\build-env-0_b2_gmq\lib\site-packages\mesonpy\_tags.py", line 83, in __init__
    f'Unknown Python implementation: {self.implementation}. '
NotImplementedError: Unknown Python implementation: mesonpy. Please report this to https://github.com/FFY00/mesonpy/issues and include information about the Python distribution you are using.

ERROR Backend subprocess exited when trying to invoke build_wheel

Funny enough, the same builds on macOS and Linux and on Windows with Python 3.8 and later. I have absolutely no clue about what may be wrong.

[dnicolodi]

Investigating a little, I see that the first issue is that the extension filename on Python 3.7 is siphash24.pyd while on Python 3.8 and later is something like siphash24.cp38-win_amd64.pyd. This seems a Meson bug probably triggered by some Python standard lib difference between Python 3.7 and later versions.

The second issue is that the unexpected file path confuses the regular expression here

meson-python/mesonpy/__init__.py

Line 102 in 5dba657

_WINDOWS_NATIVE_MODULE_REGEX = re.compile(r'^(?P<name>.+)\.(?P<tag>.+)\.pyd$')

Maybe the name and tag groups could accept a narrower charset, something like:

_WINDOWS_NATIVE_MODULE_REGEX = re.compile(r'^(?P<name>[^.].+)\.(?P<tag>[^.]+)\.pyd$')

and the match be done on the filename part only (now I have the impression that the whole file path is matched).

[dnicolodi]

If the problem is somewhere in the standard library, it may already be fixed, however, GitHub Actions are stuck at Python 3.7.9 on Windows while the current release is 3.7.15.

[dnicolodi]

Meson names the extension module that way because it uses what the sysconfig module tells him to use, see https://github.com/mesonbuild/meson/blob/e68fcac919b332c7f9469672a243d2aab1bfea0a/mesonbuild/modules/python.py#L443

In Python 3.7.9 on Windows:

python -c 'import sysconfig; v = sysconfig.get_config_vars(); print((v.get("EXT_SUFFIX"), v.get("SO"), v.get(".so")))'
('.pyd', '.pyd', None)

While with Python 3.8.10 on Windows:

python -c 'import sysconfig; v = sysconfig.get_config_vars(); print((v.get("EXT_SUFFIX"), v.get("SO"), v.get(".so")))'
('.cp38-win_amd64.pyd', '.cp38-win_amd64.pyd', None)

Digging through the Python changelogs, I found that sysconfig was fixed to behave correctly on Windows in Python 3.8.7, see https://docs.python.org/3.8/whatsnew/changelog.html#python-3-8-7-final :

bpo-39825: Windows: Change sysconfig.get_config_var('EXT_SUFFIX') to the expected full platform_tag.extension format. Previously it was hard-coded to .pyd, now it is compatible with distutils.sysconfig and will result in something like .cp38-win_amd64.pyd. This brings windows into conformance with the other platforms.

Therefore it seems that Meson gets the extension filename wrong on Python versions smaller than 3.8.7 and meson-python trips on that. Most likely something to fix in Meson.

[dnicolodi]

With mesonbuild/meson#10961 Meson generates the extension module with the correct file name: siphash24.cp37-win_amd64.pyd for Python 3.7 vs siphash24.cp38-win_amd64.pyd for Python 3.8. However, pip does not like the wheel that meson-python produces:

ERROR: siphash24-1.1.dev0-cp37-cp37-win_amd64.whl is not a supported wheel on this platform.

despite the fact that the wheel filename is in line with what works just fine on Python 3.8: siphash24-1.1.dev0-cp38-cp38-win_amd64.whl

[dnicolodi]

The correct wheel filename for CPython 3.7 is siphash24-1.1.dev0-cp37-cp37m-win_amd64.whl

[FFY00]

Thank you for looking into it, it is really appreciated! It seems like we may need to implement a workaround for this bug.

[dnicolodi]

I don't think a workaround is necessary. The extension module filename can be fixed in Meson, see the linked PR. The wheel ABI tag determined by meson-python is just wrong and need fixing, but I would not call this a workaround.

There is no way to get the correct ABI tag from just the extension filename. However, I think meson-python can assume that the interpreter used to run itself is the one it is building for, thus it can use introspection to get the correct ABI tag.

For reference, here is what wheel does to compute the ABI tag https://github.com/pypa/wheel/blob/7b9e8e1022b9c850756ef34bc1a4a88042988a86/src/wheel/bdist_wheel.py#L71

[FFY00]

I agree, we should switch to introspecting the ABI from the interpreter and simply validate that the ABI tags in native modules are compatible. Originally, I didn't think it was possible to force Meson to use a specific Python interpreter, so we couldn't trust the introspection, but we discovered that it was possible by providing a cross file, which we are doing now.