Don't run regen-sbom in regen-all - it requires internet access

[befeleme]

Bug report

Bug description:

make regen-all should be possible to run when building without the internet access. regen-sbom hits PyPI to determine pip's metadata and exits if not successful, so it shouldn't be a part of the regen-all target.

In previous cases like this, the problematic targets were removed (#112120)

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-114240: Skip verifying pip SBOM entry against PyPI if offline outside CI #114255

[erlend-aasland]

Note that make regen-configure has required internet access for quite a while; this is nothing new.

[sethmlarson]

@befeleme Apologies for the added friction! I would like to keep regen-sbom as a part of regen-all, I can make the generate_sbom.py script more resilient in the case of no internet connection and reuse pip's SBOM entry in that case.

[sethmlarson]

I've implemented my proposal in this PR: #114255

[befeleme]

Note that make regen-configure has required internet access for quite a while; this is nothing new.

It is, but it's not a part of make regen-all target, see: https://github.com/python/cpython/blob/main/Makefile.pre.in#L1370

[hroncok]

This is still not working as expected. When the build happens offline:

[cpython (main)]$ LANG=C.utf-8 make regen-sbom 
python3.13 ./Tools/build/generate_sbom.py
Packages defined in SBOM tool don't match those defined in SBOM file: {'libb2', 'pip', 'macholib', 'msgpack', 'certifi', 'resolvelib', 'tenacity', 'pyparsing', 'chardet', 'webencodings', 'packaging', 'expat', 'urllib3', 'six', 'hacl-star', 'rich', 'platformdirs', 'idna', 'typing_extensions', 'requests', 'distlib', 'tomli', 'mpdecimal', 'distro', 'truststore', 'pygments', 'colorama', 'cachecontrol', 'setuptools', 'pyproject-hooks'}, {'mpdecimal', 'libb2', 'hacl-star', 'pip', 'expat', 'macholib'}
See 'https://devguide.python.org/developer-workflow/sbom' for more information.
make: *** [Makefile:2842: regen-sbom] Error 1

This behavior was released in Python 3.12.2.

[hroncok]

The problem is from https://github.com/python/cpython/pull/114450/files#r1482058937

[hroncok]

To clarify ours (= Fedora, i.e. including @befeleme) use case:

• we run the builds offline
• we (usually) remove the ensurepip pip wheel and use our own wheel from /usr/share/python-wheels

That combination indeed works, as was confirmed by @befeleme in #114450 (comment)

However, we sometimes don't remove the ensurepip pip wheel. E.g. when we bootstrap new Python version or when the particular pip wheel version for a Fedora release is too old (or too new) for the given Python version. This happens for example with Python 3.12 on Fedora Linux 38 (our oldest still supported release). When we tried to update to 3.12.2, we had this combination:

• we run the builds offline
• we keep the ensurepip pip wheel

That is when we have experienced the error.

The same problem will likely happen when we try to update to 3.13.0a4 in Fedora Linux 38, unless fixed before the release.

[sethmlarson]

Apologies @hroncok, this would get fixed by moving the entire pip discovery process into the python/release-tools repository, something that I want to do for other reasons too.

I've created this issue to track that work.