Pip should provide a way to specify index lookup order

[stefanoborini]

What's the problem this feature will solve?

Currently, --extra-index-url will always operate after the pypi url, no matter what.
This has already been debated at length in #3454 and #5045, where it is hinted that not even by specifying --index-url takes over the order.

In these issues, the accepted solution is to use devpi, or just to use a non-taken name on pypi. However, both these solutions are workarounds:

1. not all of us can use devpi. I am personally relying on artifactory with pypi support, and in large corporate environments you can't just install whatever you want.
2. If I were to use a name that is not used on pypi, my service would break as soon as someone registers that name on pypi and puts versions that are above mine, basically taking over my installation. This is not only annoying, but also a security problem.
3. if I were to register the name on pypi (which is not possible, unless you can push something to it, possibly fake), I could leak internal information about my company's process through the naming of the entities I reserve.

Describe the solution you'd like

Pip should have an additional option to specify exactly the order in which to honor the lookup for pypi services. This will allow to preserve backward compatibility, while solving the above issues.

Alternative Solutions

Workarounds are suboptimal, fragile, potentially a security issue, and rely on solutions that might not be implementable in a large corporate environment.

Additional context

See above posted issues.

[uranusjr]

I’m not convinced with the “not be implementable” part, at least I don’t see anyone making the argument in either of the two issues you linked. --index-url always comes first, and in a corporate setting you can overwrite that, so pip never visits PyPI (and, if you want, put in PyPI as one of the --extra-index-url so pip visits PyPI only if it does not find a package on your own index). I also don’t really think the solutions are workarounds either. Do you have a concrete example? The ordering of --index-url and --extra-index-url is very explicitly specified; they are as robust as any other alternative option proposals.

[pfmoore]

Simply re-opening the same request isn't going to change the conclusion. Python packages are intended to be unique by name, so if you use a name that is taken by another package, you will get conflicts.

I understand the issue around using a name that is subsequently taken, but this is not something that should be solved at the pip level. If you want to have a means for "reserving" names or a namespace, that is a standardisation issue across all packaging tools, and in particular should be something covered by PyPI. So this needs to be raised as a proposal for a packaging interoperability PEP.

There is already some discussion on this on Discourse. I'd suggest that if you want to riase this topic, you'd need to have a discussion there - but be aware that this is potentially a very complex issue.

(One relatively simple proposal might be to have PyPI "reserve" a certain prefix for local use, similar to the way some IP addresses are guaranteed available locally. That might address the "how do I name my private packages?" problem without getting sucked into global name registry issues).

But Python's packaging is unlikely ever to change the principle that every package named foo, regardless of source, is expected to be the same project. That's built into far too much of the infrastructure to change.

I'll leave this issue open for a short while, as I don't want to close it without giving the OP time to respond, but I'm strongly of the view that there's nothing pip can or will do here, without a standard to back any change.

[pfmoore]

(If anyone can find better labels, feel free to change - "wrong project" isn't ideal, but there isn't a "needs a standard" label...)

Edit: Never mind, I added one 🙂

[stefanoborini]

I’m not convinced with the “not be implementable” part, at least I don’t see anyone making the argument in either of the two issues you linked. --index-url always comes first

Issue #5045 explicitly says that this is not the case.

[stefanoborini]

I understand the issue around using a name that is subsequently taken, but this is not something that should be solved at the pip level. If you want to have a means for "reserving" names or a namespace, that is a standardisation issue across all packaging tools, and in particular should be something covered by PyPI. So this needs to be raised as a proposal for a packaging interoperability PEP.

the problem is that packages should be namespaced somehow.

(One relatively simple proposal might be to have PyPI "reserve" a certain prefix for local use, similar to the way some IP addresses are guaranteed available locally. That might address the "how do I name my private packages?" problem without getting sucked into global name registry issues).

That would be very nice indeed.

I'll leave this issue open for a short while, as I don't want to close it without giving the OP time to respond, but I'm strongly of the view that there's nothing pip can or will do here, without a standard to back any change.

The point is that the assumption that the name is unique is fine, but it is conditional to my choice of what order I consider them unique and with highest priority in resolving a given name.

[mboisson]

@pfmoore, since there are flags such as: --only-binary <format_control>, --no-binary <format_control>, --implementation <implementation>, --abi <abi>, which allows to control pip such that it only considers source/wheels matching specific conditions, do you think flags such as --only-localversion or something similar would be reasonable ?

To address this issue, we have started tagging all of our wheels with a localversion specifier, and it works to prefer this against an equivalent upstream version, but when a new upstream version comes out, it takes over.

Having a --only-localversion <format_control> option, which would only consider wheels with a local version tag to be considered for the packages listed, would give people dealing with the current issue a way to address it without the burden of running their own index.

[notatallshaw]

Is this different from #8606?

As I mention in #8606 (comment) we have a real world example now, in uv, of a pip-like API that by default guarantees an ordering of index lookups. uv has had to implement --index-strategy because there a real world use cases where it does not work for users.

[mboisson]

It seems to be a very similar (identical?) issue, yes. My proposal was meant to be a not-so-heavy proposal as changing the index strategy for all packages, and more of a tailored approach which could still address the core of the issue (the impossibility to tell pip to not look at pypi for specific packages).