gh-113246: Updated bundled pip to 23.3.2

[sbidoul]

• Issue: Update bundled pip to 23.3.2 #113246

[sbidoul]

I'm not sure if the failing check is related or not.

[hugovk]

Please could you also update the pip version number here?

cpython/Tools/build/generate_sbom.py

Line 53 in f428c4d

include=["Lib/ensurepip/_bundled/pip-23.3.1-py3-none-any.whl"]

And then run make regen-sbom and commit the updated Misc/sbom.spdx.json.

This is brand new tooling to generate a software bill-of-materials (SBOM), and this PR may be the first test of updating it :) Tracking issue: #112302.

cc @sethmlarson

[hugovk]

We'll get this documented: python/devguide#1241

In the meantime, if you're on macOS or Linux and get this:

❯ make regen-sbom
make: *** No rule to make target `regen-sbom'.  Stop.

Run ./configure first (https://devguide.python.org/).

If you're on Windows, run python Tools/build/generate_sbom.py

[pfmoore]

Please could you also update the pip version number here?

I would suggest updating it to say

include=["Lib/ensurepip/_bundled/pip-*-py3-none-any.whl"]

(assuming that works). I don't want us to have to manually update this every pip release - it's just another manual task that could trip people up.

[miss-islington-app]

Thanks @sbidoul for the PR, and @pfmoore for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12.
🐍🍒⛏🤖

[miss-islington-app]

Thanks @sbidoul for the PR, and @pfmoore for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11.
🐍🍒⛏🤖

[pfmoore]

@sbidoul I've triggered the backport PRs, but they'll probably need fixing for the SBOM stuff. Is that something you can do?

[miss-islington-app]

Sorry, @sbidoul and @pfmoore, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4a24bf9a13a7cf055113c04bde0874186722c62c 3.12

[miss-islington-app]

Sorry, @sbidoul and @pfmoore, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 4a24bf9a13a7cf055113c04bde0874186722c62c 3.11

[bedevere-app]

GH-113253 is a backport of this pull request to the 3.12 branch.

[bedevere-app]

GH-113254 is a backport of this pull request to the 3.11 branch.

[sethmlarson]

Apologies for putting in a roadblock without the full documentation in place, I've submitted this PR to the devguide to remedy that situation.

So this PR unfortunately has left the SBOM in an invalid state, pip has been upgraded to 23.2.2 but the SBOM still lists versionInfo: 23.2.1. The change to use * would only propagate that invalid state to future upgrades as well since CI no longer fails when an update is made.

Luckily pip (unlike most vendored dependencies) is a part of a packaging ecosystem so is easier to get the version information automatically. I've created this issue to work on automating more of this process where possible.

Happy to answer any questions about the SBOM work, this is one of the challenges of trying to add security features while minimizing the impact to core developers work so I'm looking to learn more from you all as well on what would work best.

[pfmoore]

Thanks - sorry for merging without your input, as I said I assumed automerge would wait for any requested reviewers. My bad.

Ideally, I'd hope that ultimately adding a new release of pip would mean simply updating the ensurepip __init__.py and dropping the new wheel in, as it did in the past. That was what I'd intended by suggesting using the wildcard. Was it the wildcard that made the versionInfo field invalid, or is that a different edit that we didn't know to make?

Hopefully, your ultimate resolution will be to automate all of the SBOM generation from just the wheel filename. The format of a wheel filename is standardised, and you can extract the version from it, so I'd hope that would be possible (even if it needed a small amount of special-case code for ensurepip).

[sethmlarson]

@pfmoore No worries at all! I also didn't know that was a thing so now I've learned as well.

Hopefully, your ultimate resolution will be to automate all of the SBOM generation from just the wheel filename. The format of a wheel filename is standardised, and you can extract the version from it, so I'd hope that would be possible (even if it needed a small amount of special-case code for ensurepip).

This is my vision as well, potentially with an added run of make regen-sbom after dropping in the wheel so the generation can take place. Would that be acceptable as a workflow?

[pfmoore]

Would that be acceptable as a workflow?

Yep, assuming there's a Windows equivalent of that (i.e. something that doesn't rely on make). I'm bound to forget to do it a few times, and I don't know where would be a good place to document it (thinking in terms of where I look when I need to remember the process) but we can sort details like that out.

[hugovk]

So this PR unfortunately has left the SBOM in an invalid state, pip has been upgraded to 23.2.2 but the SBOM still lists versionInfo: 23.2.1.

@sethmlarson Would you like this fixed for tomorrow's 3.13a3 release? If so, maybe fix directly, and update automation later if that will take a little longer.

[sethmlarson]

@hugovk I'll submit a PR quickly, thanks for the ping :)

[sethmlarson]

Created #113262 to fix the pip SBOM metadata.

[sethmlarson]

@pfmoore @sbidoul pip SBOM metadata discovery is now automated: #113295