Debundling, Technical Debt and Responsibilities

[pradyunsg]

This is an offshoot from #9467. Go ahead and read the most recent few comments for context.

/cc @FFY00 @eli-schwartz @stefanor @doko42 @kitterma @hroncok to put this on your radar. Please feel free to add in other redistributors of pip, who might be interested in this discussion.

/cc @pypa/pip-committers to call me out if I say something wrong here.

---

First off, I know folks probably don't say this enough, but: hey, thanks for doing what you do. I genuinely do appreciate the work you do, and it's quite a thankless task to keep OSS things functioning. So, hey, thank you!

---

I've seen it stated in multiple places, by multiple different people now, that because pip has a mechanism to simplify debundling, it should work when debundled as is.

Please don't take the fact that the debundling script exists in this repository, to mean that it is somehow pip maintainers' responsibility to ensure that it'll give you something that just works. If that's the expectation being set, I'd like to remove that script from this repo, and make it clearer that it's not something that we want to deal with.

As far as I can tell, the debundling script exists because... how do I phrase this diplomatically... it was very easy for redistributors to debundle pip, get it kinda-sorta working and ship a broken pip. It is there to make a redistributor's life easier when they're debundling pip, but that can not be at the cost of making pip's maintainers' lives harder.

Quoting from our vendoring policy's debundling-script-is-here section:

However, if you insist on doing so, we have a semi-supported method (that we don’t test in our CI) and requires a bit of extra work on your end in order to solve the problems described above.

The "bit of extra work" is making sure that the debundled pip doesn't fail in any of the ways that our users use it. (I'm saying "our users" because these are users of both pip and $thing)

For years now, we've effectively been saying "don't do that please, because it'll break things. And if you do it anyway, please make sure it works OK because we don't have the resources to test all the ways it can break for you.".

And then, redistributors and said "yea, so... it broke because we didn't account for $thing, but if you do this one thing in this specific weird way, our jenga tower stays upright". And we've been accepting (even authoring) such patches because it makes our users' lives easier.

---

It'd be very easy for pip's maintainers to point to our vendoring policy, and start saying no to patches intended to make things work when pip is debundled (and to revert the ones we've merged over the years already). We're not doing that, but I'd like to get to a point where pip's maintainers don't have to worry about the issues caused by the debundling of pip.

Honestly, pip's maintainers can't be fixing / dealing with these issues - that's literally the point of that policy document. pip getting even more changes and weird type conversions, to accommodate for the various breakages due to debundling is NOT the solution here. It's a fragile workaround, can easily be refactored and makes it more difficult to maintain this code in general. It's technical debt for pip, in exchange for avoiding additional work for downstream redistributors. We've been taking this on for a while but I'd really like to stop doing that.

Pip's¹ been taking on technical debt for something that we've explicitly documented that we don't want to be taking technical debt for. We are going to have to start saying no to these patches at some point and I'm sure everyone would prefer it wasn't a thing we started doing on a random day magically. :)

¹ I hope you're smiling Paul.

---

So... I do have a few questions now.

For the redistributors who are debundling pip, could you share with us the line of reasoning that leads you to decide that you have to be debundling pip? Feel free to point to specific sections in bunch of documents. (I promise to not get into the discussion of the merits of those choices, I just wanna know what they are to better understand your situation)

For everyone:

1. What can be done so that redistributors are not pushing the costs of debundling pip over to pip's maintainers?
2. Should pip's maintainers start saying no to taking on more technical debt?
3. Any bright ideas for improving this situation? 🙃

(PS: typed with typo-happy thumbs, on a phone)
(PPS: this is definitely a "if I had more time, I'd have written a shorter letter" situation)

[hroncok]

First off, I know folks probably don't say this enough, but: hey, thanks for doing what you do. I genuinely do appreciate the work you do, and it's quite a thankless task to keep OSS things functioning. So, hey, thank you!

❤️

ccing also @encukou @torsava @frenzymadness @stratakis from our team

[eli-schwartz]

We (Arch) consistently debundle both pip and setuptools, so the referenced issue is not a problem for us. Actually, we have seen that issue and consider it a setuptools issue...

• pip has a bit of a "best of breed" approach to vendoring modules which works pretty well, and the debundling process is basically "remove the _vendor/ modules, keep the shim and you're done".
• setuptools uses its own pkg_resources.extern thing that is broken so we remove the pkg_resources.extern module, remove the extern shim, nuke everything from high orbit, and patch out every reference we can find, amounting to a very small soft-fork of setuptools/pkg_resources which natively has no vendoring code at all.
• We never tried to test devendoring pip, but not setuptools, because that's approaching the problem backwards and I would not expect it to work anyway. One might hope it would work, but one should never gamble on it working.

I don't see any rationale for devendoring pip, but not devendoring setuptools... is their packaging policy not consistent? Why is it the pip project's responsibility to add code specifically for the case where debian politics prevents timely coordination across packages?

I would prefer pip to work with distros doing devendoring, when it is the technically correct thing for pip to have the solution. Off the top of my head, I don't particularly remember any problems Arch has had in that regard, other than issues and updates to _vendor/__init__.py itself, so I'm optimistic that we can happily work together on this.

I would prefer if Debian were to solve internal Debian bureaucracy by handling it via Debian bureaucratic channels (e.g. a bug report to the setuptools package to devendor setuptools, which is then a blocker before they can devendor pip), rather than trying to solve it by patching pip for a Debian-specific need that does not result in elegant code, and apparently increasing the maintenance burden of pip. :(

[pradyunsg]

@eli-schwartz the PR referenced is for an issue that occurs only on Arch (AFAICT): #9348

If this were specific to Debian, I'd have reached out to the Debian maintainers directly. :)

[eli-schwartz]

As far as I can tell, that too is because of inconsistent devendoring, which we don't do... But in that case, mixing a distro pip and a user installed setuptools breaks.

This is related to pypa/setuptools#1383 and more generally to the golden rule "do not overwrite/override the distro packages with your own, it will break other distro packages".

But originally I only read the PR, not the issue. :)

[pradyunsg]

For the redistributors who are debundling pip, could you share with us the line of reasoning that leads you to decide that you have to be debundling pip? Feel free to point to specific sections in bunch of documents.

If you could also elaborate on this, that'd be great!

[stefanor]

@pradyunsg: Thanks for starting this discussion.

Please don't take the fact that the debundling script exists in this repository, to mean that it is somehow pip maintainers' responsibility to ensure that it'll give you something that just works.

That's not my expectation. I assume you don't test it. And I see it as Debian's responsibility to help keep this mechanism working, as long as we're taking advantage of it. I am prepared to spend several hours scratching my head and debugging those problems, every now and then.

I expect that there may be push-back on implementation details of those patches, but not to have to re-litigate the existence of the debundling support.

And then, redistributors and said "yea, so... it broke because we didn't account for $thing, but if you do this one thing in this specific weird way, our jenga tower stays upright". And we've been accepting (even authoring) such patches because it makes our users' lives easier.

In the case of #9686, you're not wrong with that description. Finding the cause of the debundling issues that lead to it took a few days (on and off). I see @kitterma was previously aware of that issue years ago, but if I was, I'd forgotten all about it... However, these are fairly minor bugs, once understood. The Jenga tower stands fairly well...

For the redistributors who are debundling pip, could you share with us the line of reasoning that leads you to decide that you have to be debundling pip?

Debian Policy: https://www.debian.org/doc/debian-policy/ch-source.html#embedded-code-copies
Debian Wiki Page: https://wiki.debian.org/EmbeddedCopies

What are the rationales behind it? Part if it is the definition of a distribution. We collect together software and try to fashion it into a cohesive system. The fewer copies of each thing that's in it, the easier our job is, and the smaller and simpler the system is. That's our security team's view too. They want to minimize the amount of work to do in response to issues.

If we need to patch something in a library, it's really annoying to have to do it in multiple copies of that library. Especially if different people are responsible for maintaining each one. Hopefully that's not necessary, of course.

When a project is targetting a specific version of a library that is out of date and we want to replace it, we'll (often) help them to port their software to the new version. A big distro is full of many dead & semi-dead upstreams, so there is an endless amount of this work to do. (That can be a good hint that a project has died and it's package should be removed.)

Of course, every policy of course has exceptions. Here's the documented list of known embedded copies in Debian, that's far from complete.

Basically, we debundle, where we can. When we can't, we may disable the relevant features, or just carry the embedded copy (distastefully). For web browsers, for example, in practice we have to carry embedded copies of several dependencies to support our stable releases. That's obviously the right trade-off to make in that situation.

Maybe we should be doing the same thing with pip, now that the Python distribution space is evolving faster. Pip is pretty much a user-facing leaf package (not a library being used by other packages). So, we could, be shipping updated pip to stable Debian releases. And using bundled dependencies does make that very easy. This is certainly a conversation we can have.

Stable distributions are trying to offer stability, by changing as little as possible. That usually means not shipping updated versions of software, because the updates bring unexpected change. When a project has stabilised enough, and has the testing and engineering resources to reliably produce stable versions that work across a range of platforms, and won't cause too many behavioural regressions, then shipping updates becomes feasible.

It's all a matter of weighing risks to our users.

1. What can be done so that redistributors are not pushing the costs of debundling pip over to pip's maintainers?

I don't know how much space there is there. Distributors can carry the cost of writing patches, but they'll need review. And there may be knock on technical debt. I'm not aware of much of that in pip related to debundling, but maybe you can educate me on that.

Supporting debundled use (where the versions may not be exactly what you expect) encourages you to keep libraries at more arms-length from each other, rather than tight integration. I see that as good library design.

3. Any bright ideas for improving this situation? upside_down_face

I think the situation is pretty good. pip supports a common use-case for distributor modification directly upstream. We could take that one step further with upstream CI.

---

@eli-schwartz:

pip has a bit of a "best of breed" approach to vendoring modules which works pretty well, and the debundling process is basically "remove the _vendor/ modules, keep the shim and you're done".

💯

Why is it the pip project's responsibility to add code specifically for the case where debian politics prevents timely coordination across packages?

From my PoV my PR makes pip more robust when devendored. It's not pip's responsibility to take it, but if pip wants it, I'm offering it.

I like to be able to carry the smallest patch-set possible in Debian, it makes makes our life easier, and means less chance for users to experience something different to what the upstream expects. As distributors we are trying to serve both users and upstreams.

[pradyunsg]

IIRC, we had CI for debundling at some point in the past and we removed it after some discussion. The basic line of reasoning there was (1) our CI setup has wayyyy too many long-ish jobs already (2) that CI can only cover one aspect of the situation and it cannot ensure all the potential combinations of patched/unpatched pip/pip's dependencies work.

(I appreciate the responses here; but I can't respond to those yet because I have to make and eat breakfast)

[uranusjr]

Coming purely from the technical side, I wonder if it’s possible to configure Mypy to check for this. Maybe a conditional import to tell pip._vendor.packaging.version.Version and pkg_resources.Version are not interchangable, even though the latter is an alias at runtime? This would be able to ensure like 90% correctness with pretty neglectable CI addition.

[pradyunsg]

It's not pkg_resources.Version, but pkg_resources.extern.packaging.version.Version (which is removed and converted to pip._vendor.packaging.version.Version by our vendoring logic). Debundling pip and setuptools means that it's now possible that they no longer match.

[uranusjr]

Yes, and I’m thinking maybe it’s possible to “fake” that mismatch for the mypy check.

[pradyunsg]

For mypy, they both come from the exact same place: pip._vendor.packaging.version.Version in this repository. Whatever we do to convince mypy to prevent us from "mixing the same type" will be a hack, but... if it's not too horrible, color me interested! :)