setuptools_scm 7.0.1 fails to compile with --no-binary :all:

[tjwalton]

Run pip install "setuptools_scm[toml]==7.0.1" --no-binary :all: and it fails with a recursion error.
The same command pip install "setuptools_scm[toml]==6.4.2" --no-binary :all: with the previous version of setuptools_scm succeeds.
This was tested on Python 3.9 (see docker reproduction below).

I am aware that the pip documentation for --no-binary says Note that some packages are tricky to compile and may fail to install when this option is used on them but this used to work and no longer does.

$ docker run -it --rm --entrypoint=/bin/bash python:3.9
root@6dc0609a5bd3:/# pip3 install "setuptools_scm[toml]==7.0.1" --no-binary :all:  
Collecting setuptools_scm[toml]==7.0.1
  Downloading setuptools_scm-7.0.1.tar.gz (67 kB)
     ���������������������������������������� 67.9/67.9 KB 7.4 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [116 lines of output]
      Collecting setuptools>=45
        Downloading setuptools-62.6.0.tar.gz (2.6 MB)
           ���������������������������������������� 2.6/2.6 MB 45.8 MB/s eta 0:00:00
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Installing backend dependencies: started
        Installing backend dependencies: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting tomli>=1.0
        Downloading tomli-2.0.1.tar.gz (15 kB)
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting packaging>=20.0
        Downloading packaging-21.3.tar.gz (84 kB)
           ���������������������������������������� 84.8/84.8 KB 20.2 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting typing_extensions
        Downloading typing_extensions-4.2.0.tar.gz (42 kB)
           ���������������������������������������� 43.0/43.0 KB 10.9 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'done'
        Getting requirements to build wheel: started
        Getting requirements to build wheel: finished with status 'done'
        Preparing metadata (pyproject.toml): started
        Preparing metadata (pyproject.toml): finished with status 'done'
      Collecting importlib_metadata
        Downloading importlib_metadata-4.11.4.tar.gz (44 kB)
           ���������������������������������������� 44.2/44.2 KB 6.4 MB/s eta 0:00:00
        Installing build dependencies: started
        Installing build dependencies: finished with status 'error'
        error: subprocess-exited-with-error
      
        × pip subprocess to install build dependencies did not run successfully.
        │ exit code: 2
        ╰─> [59 lines of output]
            Collecting setuptools>=56
              Using cached setuptools-62.6.0.tar.gz (2.6 MB)
              Getting requirements to build wheel: started
              Getting requirements to build wheel: finished with status 'done'
              Installing backend dependencies: started
              Installing backend dependencies: finished with status 'done'
              Preparing metadata (pyproject.toml): started
              Preparing metadata (pyproject.toml): finished with status 'done'
            Collecting setuptools_scm[toml]>=3.4.1
              Using cached setuptools_scm-7.0.1.tar.gz (67 kB)
            ERROR: Exception:
            Traceback (most recent call last):
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/cli/base_command.py", line 167, in exc_logging_wrapper
                status = run_func(*args)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/cli/req_command.py", line 205, in wrapper
                return func(self, options, args)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/commands/install.py", line 339, in run
                requirement_set = resolver.resolve(
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/resolver.py", line 94, in resolve
                result = self._result = resolver.resolve(
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 481, in resolve
                state = resolution.resolve(requirements, max_rounds=max_rounds)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 348, in resolve
                self._add_to_criteria(self.state.criteria, r, parent=None)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_vendor/resolvelib/resolvers.py", line 172, in _add_to_criteria
                if not criterion.candidates:
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
                return bool(self._sequence)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
                return any(self)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
                return (c for c in iterator if id(c) not in self._incompatible_ids)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
                candidate = func()
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/factory.py", line 215, in _make_candidate_from_link
                self._link_candidate_cache[link] = LinkCandidate(
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 288, in __init__
                super().__init__(
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 158, in __init__
                self.dist = self._prepare()
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 227, in _prepare
                dist = self._prepare_distribution()
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/resolution/resolvelib/candidates.py", line 299, in _prepare_distribution
                return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/operations/prepare.py", line 487, in prepare_linked_requirement
                return self._prepare_linked_requirement(req, parallel_builds)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/operations/prepare.py", line 556, in _prepare_linked_requirement
                dist = _get_prepared_distribution(
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/operations/prepare.py", line 57, in _get_prepared_distribution
                with req_tracker.track(req):
              File "/usr/local/lib/python3.9/contextlib.py", line 119, in __enter__
                return next(self.gen)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 122, in track
                self.add(req)
              File "/tmp/pip-standalone-pip-c_aqwbg6/__env_pip__.zip/pip/_internal/req/req_tracker.py", line 92, in add
                raise LookupError(message)
            LookupError: https://files.pythonhosted.org/packages/2a/b8/c9256baee670a8e67c9ea565869dcc75fc8fcaa369bb9b8d5a1bfe301985/setuptools_scm-7.0.1.tar.gz#sha256=d1c631fc3a4281754274155ef4a40de24b1d1c718df00ead147b08ab690615b2 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.7) is already being built: setuptools_scm[toml]==7.0.1 from https://files.pythonhosted.org/packages/2a/b8/c9256baee670a8e67c9ea565869dcc75fc8fcaa369bb9b8d5a1bfe301985/setuptools_scm-7.0.1.tar.gz#sha256=d1c631fc3a4281754274155ef4a40de24b1d1c718df00ead147b08ab690615b2
            WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
            You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: subprocess-exited-with-error
      
      × pip subprocess to install build dependencies did not run successfully.
      │ exit code: 2
      ╰─> See above for output.
      
      note: This error originates from a subprocess, and is likely not a problem with pip.
      WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
      You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0.4; however, version 22.1.2 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

[cho-m]

We hit this issue in Homebrew too.

It appears to be due to changes to support Python 3.7 (d3d2573) which has created a build loop of setuptools_scm --> importlib_metadata --> setuptools_scm[toml]
https://github.com/pypa/setuptools_scm/blob/f807bfe088a756113ecfe9c2d3d57932a2e5e623/pyproject.toml#L7

For Python 3.8+, one option is to remove build dependency on importlib_metadata inside pyproject.toml to match setup.cfg, e.g.

    "importlib_metadata; python_version < '3.8'",

Not sure if possible to break circular dependency in Python 3.7.

[RonnyPfannschmidt]

I think it's doable, however I have to note that your target build process is completely broken and needs a complete overhaul

[iMichka]

I have to note that your target build process is completely broken and needs a complete overhaul

Homebrew's? Maybe. I would not say 'completely' broken, but rather fragile :) But we can only improve, it's just a question of focus / maintainer bandwidth and coordination (which is hard).

Let's not diverge from the initial topic (unless the broken Homebrew build process has an impact here).
I feel the issue is also present outside of Homebrew, as the first report in this issue does not talk about Homebrew.
Also, I saw someone adding a comment to the commit linked by @cho-m : d3d2573#r76673659

[RonnyPfannschmidt]

the oversight in the linked commit simply elevates the bootstrap problem from python3.7 only to all supported python

the issue with homebrew however also is - that the linked build processes intentionally use a unsupported process that is known to trigger bootstrapping issues that need extra care

in a normal build, the isolated and used environments can gracefully just go with the released wheel artifacts which are install-able

we do have some self-bootstrap testing, but that assumes basic compliance with common python processes, homebrew does not comply, and as per the warning now things fall over

making things work on python3.8+ will be easy as all thats missing is a environment marker,
enabling bootstrap on python3.7 itself is not a priority as the supporting artifacts (aka pure python wheels) are available

[RonnyPfannschmidt]

when removing the dependencies for a local test, im running into pypa/setuptools#2204

so as far as i gather this one is not fixable for no-binary builds in a quick manner

[RonnyPfannschmidt]

im working on a possible fix by removing the build time requirement for importlib metadata