index-url extra-index-url install priority order

[njourdane]

This is totally related to #5045, but I can not comment on it.

For me this should be re-opened, because it's a security issue.

I provide the package xxx to the private repository my-company.com. Then I gives installation instructions to collaborators, configure CI to install this package, etc.

The problem is if someone add a package with the same name on pypi.org, then users and CI will get this package instead of mine, which is bad: and I want to be sure that the installation will not change.

It's not about naming package: if I name my packe my_company.xxx, a malicious user could chose the exact same name and publish it to pypi, letting my collaborators installing a malware instead of my package.

[pfmoore]

This is how index behaviour and package uniqueness is defined. It's not a security issue. If you don't trust PyPI, you should exclude it altogether from your package index searches.

[njourdane]

Yes but Pypi is used to download my project dependencies and I don't want to include all of them in our private repo.

[pfmoore]

Understood - so you're trusting PyPI, for better or worse. I don't have a solution for you, but treating all indexes as equal is by design. So if you want a solution for your use case, you need to describe what you'd want to happen as a new feature, without changing existing behaviour. It can then be considered - but would likely need someone to submit a PR once the approach is agreed.

[uranusjr]

There is a discussion on the philosophy around this design choice: https://discuss.python.org/t/4045

In concrete words, pip is consistently treating pypi.org as a global namespace registry, while your mental model is treating it as a package repository. This is still an actively debated topic, but at this point neither view is “wrong”. To use this feature as pip designs it, you should name-squat on PyPI to get what you want.

[pfmoore]

Philosophical issues to one side, I'd note that devpi fixes this problem, and is available as a solution right now. From the documentation of "index inheritance":

All privately uploaded packages will by default inhibit lookups from pypi, allowing to stay safe from an attacker who could otherwise upload malicious release files to the public PyPI index.

I don't use devpi myself, so I don't know the details, but I would strongly recommend that @roipoussiere and anyone else who feels that they would like "priority order" for indexes in pip, look at whether devpi solves their issue.

(Yes, I know it has the problem of "but it means I have to set up a server in my local environment". I have some sympathy for this, as I work in an environment where installing local servers is frustratingly painful thanks to internal policies and bureaucracy. But conversely, please remember that asking a group of volunteer maintainers to make significant code changes to address an issue that already has a solution doesn't come without cost either...)

[njourdane]

Thank you for these detailed responses.

So if you want a solution for your use case, you need to describe what you'd want to happen as a new feature, without changing existing behaviour.

Ok I understand. Here my suggestion:

Create a new pip boolean option prefer-extra, defaulted to False. If set to True, pip will look for the packages in the repositories defined in extra-index-url, in the definition order.

[pfmoore]

-1 (you shouldn't be too surprised to hear...)

It's way too easy for people to set a flag in config and forget they have it set, and then be confused when legitimate upgrades don't work. Or we'd get people wanting to set it in requirements files, but its effect wouldn't be limited to the requirement file. Also, the proposal doesn't explain how this would interact with --find-links.

I don't think you fully understood my reservations - this is not a simple thing to change, and it will take some work to flesh out a viable proposal.

I did a quick test with devpi, and setting up a devpi server with your local packages stored in a private index backed by the default PyPI mirror index does precisely what you want. Given that devpi solves this problem out of the box, I'm going to be very hard to convince that this needs a solution within pip.

[ivanst0]

@roipoussiere, have you considered using this setup?
--index-url <your private feed> --extra-index-url https://pypi.org/simple
I believe this would make pip search for a package in your private feed first and then in the public repository.

[pfmoore]

@ivanst0 It doesn't. The indexes are treated equally.

[jsiverskog]

https://www.bleepingcomputer.com/news/security/researcher-hacks-over-35-tech-firms-in-novel-supply-chain-attack/