Rationalise and improve the `--target`, `--prefix` and `--root` options

[pfmoore]

What's the problem this feature will solve?

At present, pip allows the user to select alternative install locations using --target, --prefix and --root. However, these are install-only options, and so do not work for other pip commands, like list or uninstall.

Describe the solution you'd like

Make the "target install scheme" options global, so that they apply to all commands.

In particular, be able to manage directories populated using pip install --target the same as a site-packages directory (with support for upgrades, uninstalls, list and list --outdated, etc).

Alternative Solutions

Stick with what we have. Continue requiring users to delete and recreate directories populated via --target whenever they want to change things.

Additional context

In terms of implementation, experiments suggest that this can be achieved just by modifying get_scheme in pip._internal.locations, and get_(default)_environment in pip._internal.metadata. In practice, the bulk of the work will be removing the existing code in favour of the new approach.

The proposed new approach will not work with the "legacy" setup.py install mechanism. We will need to decide whether to continue to support the existing options in that case. If we do, care will be needed to leave "just enough" of the existing code to support that route. The alternative would be to desupport using these options with the legacy install route, either by failing with an error suggesting use of --use-pep517, or by simply switching that option on automatically. Or we could delay implementing this feature until the legacy install route is finally removed 🙂

A feasible alternative/transition method would be to add one or more new global options that replicated the behaviour of the existing options using the new approach. The old options could then be removed at a later date. The big issues here are (1) thinking of good names for the "new" option(s), and (2) replicating all of the existing tests to check the new approach as well as the old one.

One further point to consider is that --target currently appears to install into a temporary directory and move the files into the target location once the install is complete. It's not clear why this is done, and I could find no documentation that either explained or guaranteed this behaviour. Would it be acceptable to remove that as part of the reorganisation? (My best guess is that it's to avoid a failed install damaging the target location - if we make --target do a "first class" install, with upgrade and uninstall support, our existing protections against breaking environments should be a superior replacement for the "temporary location and copy" approach).

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Or we could delay implementing this feature until the legacy install route is finally removed 🙂

That’s what I’d prefer.

[pfmoore]

Do we have any feel for how long it's likely to be before we get to that point? I know @sbidoul has been working on this, but I don't have a good feel for timescales.

[sbidoul]

I've restarted work on deprecating all setup.py install code paths. And I'm making good progress. #11331 is ready to merge. And I think I finally found the elusive good path to disentangle --no-binary (the first step is ready to review in #11359).

So if all goes well, by the end of this year, the deprecation of setup.py install will be explicit for all use cases, and there will be alternate mechanisms for users (possibly depending on pypa/setuptools#2491 and #11325).

After that, it will be a matter of getting a feel of user impact to decide when to switch to --use-deprecated and finally pull the plug. That is harder to predict.

Regarding this proposal: +100. On the how, I'd be fine if pip aborts when hitting setup.py install with the new scheme.

[septatrix]

So which one of these should one currently use to install packages into another root directory, e.g. a chroot? --root "$BUILDROOT" --prefix /usr? Should --target also be used? And what exactly is the difference between python3.11 -m pip and python3.12 -m pip --python python3.11?

[ichard26]

I'd like to chip away at #9081, but before I can do that, I need to understand all of the pip install options. I've never used the --root or --prefix flags, and I've rarely used --target. I'd appreciate if someone could explain what these options do and when/why I'd use each one.¹ Briefly checking the code, there's also a home directory, whatever that is.

I'll do my own research/exploration, but a quick overview would be helpful!

Footnotes

1. In practice, I doubt the build process should respect these options, but it'd be good to understand them regardless. ↩

[septatrix]

I'd like to chip away at #9081, but before I can do that, I need to understand all of the pip install options. I've never used the --root or --prefix flags, and I've rarely used --target. I'd appreciate if someone could explain what these options do and when/why I'd use each one.

The --root option is useful to install the package into a filesystem different from your own machine (/). This can be very useful when building (minimal) images and containers when you want to install things but don't want to or can't (because e.g. pip is not installed) run the tools from inside of them.

When using --root, the prefix defaults to /usr/local which is where traditionally you would install software that you build/installed yourself and that is not provided by the distribution. In the case of custom OS images and containers this separation is not really needed/wanted. Therefore you generally choose --prefix /usr or in some legacy cases --prefix / but that has several drawbacks.

--target, however, I have not really used yet nor do I understand what exactly it does

[sbidoul]

I doubt the build process should respect these options

Actually I'd worry if we discover these options have any influence whatsoever on the build process.

[mason3k]

One further point to consider is that --target currently appears to install into a temporary directory and move the files into the target location once the install is complete. It's not clear why this is done, and I could find no documentation that either explained or guaranteed this behaviour. Would it be acceptable to remove that as part of the reorganisation?

I don't pretend to know all the considerations here, but I do think removing this would actually fix #12113. The passage through the temp directory causes the permissions from the temp directory to 'stick' on Windows instead of inheriting permissions from the new target directory.

[mason3k]

Don't know if this is typical, but FWIW I've used --target for installing some large shared dependencies to a directory and then making that directory accessible in multiple venvs that share those resources with a .pth file entry