[Request for Reverting Intentional Breaking Change] New license file validation breaks projects with non-standard layout

[tiran]

setuptools version

77.0.1

Python version

any

OS

any

Additional environment information

No response

Description

The new license file validation logic from #4838 breaks projects that have a non-standard layout. Projects like Apache Arrow or Triton don't have their Python code in the root directory of the sources. Instead it is in a subdirectory, typically called python. For example Apache Arrow has a python/setup.cfg with

[metadata]
license_files =
  ../LICENSE.txt
  ../NOTICE.txt

Expected behavior

setuptools does not break software that uses a non-standard layout and reference license files from a parent directory.

How to Reproduce

1. download and unpack https://github.com/apache/arrow/archive/refs/tags/apache-arrow-19.0.1.tar.gz
2. cd arrow-apache-arrow-19.0.1/python
3. python3 -m venv venv
4. . venv/bin/activate
5. pip install -U setuptools cython
6. python3 setup.py

Output

Traceback (most recent call last):
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/setup.py", line 399, in <module>
    setup(
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/__init__.py", line 116, in setup
    _install_setup_requires(attrs)
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/__init__.py", line 87, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/dist.py", line 752, in parse_config_files
    self._finalize_license_files()
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/dist.py", line 466, in _finalize_license_files
    self.metadata.license_files = list(unique_everseen(files))
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/_vendor/more_itertools/recipes.py", line 511, in unique_everseen
    for element in iterable:
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/dist.py", line 483, in <genexpr>
    for path in sorted(cls._find_pattern(pattern, enforce_match))
  File "/home/cheimes/tmp/arrow-apache-arrow-19.0.1/python/venv/lib64/python3.9/site-packages/setuptools/dist.py", line 506, in _find_pattern
    raise InvalidConfigError(f"Pattern {pattern!r} cannot contain '..'")
setuptools.errors.InvalidConfigError: Pattern '../LICENSE.txt' cannot contain '..'

[cdce8p]

Seems the PEP 639 logic is applied to the old metadata.license_files field. It might make sense to fix it for existing configs but do note that it's invalid according to the spec.

Parent directory indicators (..) MUST NOT be used.

https://peps.python.org/pep-0639/#add-license-files-key

[cdce8p]

I'm not sure we're able to differentiate where the license file pattern is coming from currently as both get merged into self.metadata.license_files. The best we might be able to do would probably be to change the error to a DeprecationWarning.

@abravalheri What do you think?

setuptools/setuptools/dist.py

Lines 507 to 510 in 62d4512

	if pattern.startswith((os.sep, "/")) or ":\\" in pattern:
	raise InvalidConfigError(
	f"Pattern {pattern!r} should be relative and must not start with '/'"
	)

[abravalheri]

It seems that these projects are not going to be compatible with the PEP. So the only options are:

1. The projects pin Setuptools < 77
2. We change to a deprecation warning as suggested by @cdce8p and we give these projects sometime to rethink the strategy.

What puzzles me is that setuptools should not allow files from outside the project root to be included... I don't know which technique these projects are using to get around that (are they skipping the sdist phase?).

[rizenfrmtheashes]

For what it's worth, PreCommit with pylint is very broken right now.

Pre-Commit doesn't give you any kind of options to restricting the versions of sub-dependencies, and with no upper-dependency bound on pylint for setuptools, one of its dependencies violates this license check and pre-commit will always fail.

Ideally, it would be nice if we changed this to a warning so packages have some time to make short term changes to pin setuptools to a version before 77.x, and long term make changes to fix their license file issues as appropriate.

[oerp-odoo]

Not sure if this is related, but now our CI is broken when latest setuptools (77.0.1) is installed, with this error:

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]

Reverting back to 76.0.0, fixes it.

[abravalheri]

Not sure if this is related, but now our CI is broken when latest setuptools (77.0.1) is installed, with this error

@oerp-odoo does not seem to be related. setuptools is not available in the build environment seems to indicate that setuptools is not even installed, so there is no part of setuptools code running yet to be able to cause a bug. Please make sure you have a proper pyproject.toml file with the [build-system] section configured. If you have a minimal reproducer that explains how setuptools is implied¹, please open a new issue.

Footnotes

1. So far, I don't think setuptools is causing this problem, it does not even seem to be installed. ↩

[Togtja]

We were getting on one of our Python projects

from packaging.licenses import canonicalize_license_expression
ModuleNotFoundError: 
No module named 'packaging.licenses' 

We have been using a specific version of packaging for a long time so we didn't think it was any changes there, but we were using the newest setup tools and saw that an update had happend the day before, so going back a single version fixed it

[abravalheri]

I'm not sure we're able to differentiate where the license file pattern is coming from currently as both get merged into self.metadata.license_files. The best we might be able to do would probably be to change the error to a DeprecationWarning.

So I have been doing some experiments, it turns out the solution is more complex than simply replacing the error for a warning:

• The PEP also require translating the relative paths inside .dist-info/licenses/ directory. So that .. might break things up.
• We cannot guarantee other tools in the ecosystem (e.g. package uploader, package index, installers) will not raise an exception when encountering invalid License-File in METADATA v2.4.

It also seems that this approach never actually worked 100% and sometimes it was silently producing distributions that do not contain the license files, as we can see in:

• the "log warning" in the build logs for Apache Arrow: https://github.com/apache/arrow/actions/runs/13928232961/job/38978388941#step:6:2595
• the absence of license files in the published sdist: https://inspector.pypi.io/project/pyarrow/19.0.1/packages/7f/09/a9046344212690f0632b9c709f9bf18506522feb333c894d0de81d62341a/pyarrow-19.0.1.tar.gz/

So there seems to be 1 option to be driven by setuptools: in addition to converting the error into warning, we also ignore the invalid glob completely. This should produce valid metadata and is likely to pass further validations of other tools, however it will not include the files (a similar behaviour to what is already happening in the CI logs and sdist on PyPI).

I am not sure if this is the best approach as package authors might want to ensure to distribute license information. In that case the best action would be for the packages to accommodate the new PyPA standard (or freeze the setuptools version while they don't manage to do that).

@tiran, @cdce8p what are your thoughts?

If anyone has an alternative solution, please feel free to submit a PR and I will be happy to review.

[oerp-odoo]

Not sure if this is related, but now our CI is broken when latest setuptools (77.0.1) is installed, with this error

@oerp-odoo does not seem to be related. setuptools is not available in the build environment seems to indicate that setuptools is not even installed, so there is no part of setuptools code running yet to be able to cause a bug. Please make sure you have a proper pyproject.toml file with the [build-system] section configured. If you have a minimal reproducer that explains how setuptools is implied1, please open a new issue.

Footnotes

1. So far, I don't think setuptools is causing this problem, it does not even seem to be installed. [↩](#user-content-fnref-1-9b82e8c62d2a6c2f673b45f3b89d5913)

But the problem is only occurring with 77.0.1 version. And it started today. Yesterday everything was working fine installing packages on CI.

We use https://github.com/py-actions/py-dependency-install and it has update-setuptools: "true" option, which makes sure setuptools is up to date. I also tried explicitly installing 77.0.1, but was getting the same error.

Basically:
This fails

pip install setuptools==77.0.1
pip install -r requirements.txt

This does not fail:

pip install setuptools==76.0.0
pip install -r requirements.txt

Though not sure why it fails with latest setuptools