[BUG] setuptools version 73.0.0 breaks Pythran

[agriyakhetarpal]

setuptools version

setuptools==73.0.0 (latest released version available on PyPI)

Python version

All supported ones

OS

any

Additional environment information

This currently occurs with the latest release of Pythran (version 0.16.1 as released on PyPI on 28/05/2024), and older versions have not been checked against.

Description

Hi there! The latest release of setuptools currently breaks SciPy' builds (scipy/scipy#21416) and, in turn, SciPy's recipe in Pyodide (pyodide/pyodide#5018) because SciPy now uses Pythran as a required dependency for transpiling Python to C++ (as reported in serge-sans-paille/pythran#2228).

The truncated output from Pythran is as follows – detailed output(s) below:

TypeError: metaclass conflict: the metaclass of a derived class must be a       
(non-strict) subclass of the metaclasses of all its bases

I have bisected the problem between the v72.2.0 and v73.3.0 tags as reported in scipy/scipy#21416 (comment), and the breakage seems to be coming from #4503 (i.e., f9398b1). I have opened this issue for visibility while I continue to investigate the problem at the time of writing.

Also, please refer to @rgommers's comments in serge-sans-paille/pythran#2228 (comment) that describe the situation in context to SciPy.

Expected behavior

The expected behaviour is that Pythran works and can be used to compile extension modules, whether for SciPy or elsewhere for all recent-enough versions of setuptools, since setuptools is a dependency of Pythran.

A workaround for this is available via serge-sans-paille/pythran#2229 which pins setuptools to <73, and I have another hotfix ready at serge-sans-paille/pythran#2230 which passes the tests.

How to Reproduce

Here is a minimal reproduction script:

# Set up the Pythran repository
git clone https://github.com/serge-sans-paille/pythran.git pythran/
cd pythran/
# Activate a virtual environment, install the project, etc.
python3.X -m virtualenv venv
source venv/bin/activate
pip install -e ."[doc,test]"
# Run the test suite (can take long, running just one file works as well)
python -m pytest pythran/tests/test_list.py

Output

❯ python -m pytest pythran/tests/test_list.py      
============================================================================ test session starts ============================================================================
platform darwin -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: /Users/agriyakhetarpal/Desktop/pythran
configfile: pyproject.toml
plugins: nbval-0.11.0, xdist-3.6.1
collected 0 items / 1 error                                                                                                                                                 

================================================================================== ERRORS ===================================================================================
________________________________________________________________ ERROR collecting pythran/tests/test_list.py ________________________________________________________________
/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:995: in exec_module
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
pythran/tests/__init__.py:27: in <module>
    from pythran import compile_pythrancode, spec_parser, load_specfile, frontend
<frozen importlib._bootstrap>:1412: in _handle_fromlist
    ???
pythran/__init__.py:127: in __getattr__
    import pythran.toolchain
pythran/toolchain.py:11: in <module>
    from pythran.dist import PythranExtension, PythranBuildExt
pythran/dist.py:142: in <module>
    class PythranBuildExt(PythranBuildExtMixIn, LegacyBuildExt, metaclass=PythranBuildExtMeta):
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
========================================================================== short test summary info ==========================================================================
ERROR pythran/tests/test_list.py - TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================= 1 error in 0.40s ==============================================================================
venv(base) 
pythran on  master [$] via 🐍 v3.12.4 (venv) 
❯ uv pip install "setuptools>=73"
Resolved 1 package in 16ms
Uninstalled 1 package in 0.80ms
Installed 1 package in 13ms
 - setuptools==72.2.0.post20240820 (from file:///Users/agriyakhetarpal/Desktop/setuptools)
 + setuptools==73.0.0
venv(base) 
pythran on  master [$] via 🐍 v3.12.4 (venv) 
❯ python -m pytest pythran/tests/test_list.py
============================================================================ test session starts ============================================================================
platform darwin -- Python 3.12.4, pytest-8.3.2, pluggy-1.5.0
rootdir: /Users/agriyakhetarpal/Desktop/pythran
configfile: pyproject.toml
plugins: nbval-0.11.0, xdist-3.6.1
collected 0 items / 1 error                                                                                                                                                 

================================================================================== ERRORS ===================================================================================
________________________________________________________________ ERROR collecting pythran/tests/test_list.py ________________________________________________________________
/opt/homebrew/Cellar/python@3.12/3.12.4/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:995: in exec_module
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
pythran/tests/__init__.py:27: in <module>
    from pythran import compile_pythrancode, spec_parser, load_specfile, frontend
<frozen importlib._bootstrap>:1412: in _handle_fromlist
    ???
pythran/__init__.py:127: in __getattr__
    import pythran.toolchain
pythran/toolchain.py:11: in <module>
    from pythran.dist import PythranExtension, PythranBuildExt
pythran/dist.py:142: in <module>
    class PythranBuildExt(PythranBuildExtMixIn, LegacyBuildExt, metaclass=PythranBuildExtMeta):
E   TypeError: metaclass conflict: the metaclass of a derived class must be a (non-strict) subclass of the metaclasses of all its bases

[agriyakhetarpal]

Thanks for linking the fix, @Avasam. Having taken a look, the conflict arose because PythranBuildExt is trying to use two different metaclasses:

• PythranBuildExtMeta (explicitly specified), and
• ABCMeta (inherited from Command via LegacyBuildExt)

IIUC, a class can only have one metaclass, and when inheriting from multiple classes with different metaclasses, those metaclasses must be compatible (i.e., one must derive from the other).

[Avasam]

@jaraco If you feel like reverting my change for a quick fix, go ahead. It didn't intend to cause metaclass issues. Otherwise a PR should already be ready: #4580

@agriyakhetarpal Yep, that's what I found as well through my investigation. Thanks for the easy reproducer.
I think class PythranBuildExtMeta(ABCMeta, type) would technically work, but it was definitely not the intention to force downstream users to update their metaclasses and that was an oversight. The intention was to prevent instantiating w/o subclassing from a type-checker's pov only.

No change should be needed from Pythran

[agriyakhetarpal]

Thank you for the speedy fix! I don't know if you can keep something like class Command(_Command, metaclass=ABCMeta) and make it use the metaclass directly instead of making Command inherit from ABC, so that your original changes can be retained in another way? Then, @abstractmethod could be used without changing the class hierarchy.

Edit: or maybe it is better to not over-complicate things, and the issue has now been closed just as I edited my comment 😅

[Avasam]

I don't know if you can keep something like class Command(_Command, metaclass=ABCMeta) and make it use the metaclass directly instead of making Command inherit from ABC,

Unfortunately not (and I just tested all 3 locally just to validate what I'm saying is true). ABC is a shorthand for metaclass=ABCMeta because it's just: class ABC(metaclass=ABCMeta). In the end you end up with the same problem.

@abstractmethod should be usable independently. It's just that type-checkers won't know that a class is meant to be abstract if it has no @abstractmethod. It's just a shortcoming of python's type system often relying on decorators, which are not applicable to non-function attributes.

[agriyakhetarpal]

Thanks for the explanation, @Avasam!

[abravalheri]

@agriyakhetarpal , adding a metaclass and trying to combine it with an already existing class hierarchy that is part of a 3rd party package sounds like a very fragile approach and prone to errors.

I don't know if that could work, but something a bit more flexible would be:

class PythranBuildExtMeta(type(Command)):
  ...

But I haven't tested, and I don't know if that would work.

If that works, maybe it would give us the chance to re-revert back to ABC in a future release...

[agriyakhetarpal]

Yes, @abravalheri, that makes sense. I was trying to do something similar to yours in serge-sans-paille/pythran#2230, copying over the diff from there:

class PythranBuildExtMeta(type(LegacyBuildExt)):
    def __getitem__(self, base):
        return type(
            'PythranBuildExt', (PythranBuildExtMixIn, base), {})

to work around the conflict, where LegacyBuildExt is just setuptools.command.build_ext (if available, or the distutils equivalent).

Note: not an expert on metaclasses, so I could, of course, be wrong with my experiments 😄

[abravalheri]

If that works, that could be a more stable/preferred long-term solution, and the PR would be worth pursuing even after the patch that @Avasam provided.