Prepare for the future of distutils-in-setuptools

[pradyunsg]

At least one of the breakages induced by setuptools 50.0 is already confirmed as a pip bug, and there very well may be more. pip needs a detailed (if not complete) assessment to its setuptools compatibiliy and fix the compatibility issues so the world does not break again when the distutils switch eventually happens.

Originally posted by @uranusjr in #8824 (comment)

Consider this issue a discussion point for that assessment and pip's preparedness for the same.

[pfmoore]

One thing I'd like to propose is that we avoid using distutils functions from setuptools. We've worked hard to decouple pip from setuptools - it would be bad to now re-couple the two. We don't want to depend on a build backend just for some utility functions.

A quick scan of the pip codebase shows the following imports:

.\src\pip\_internal\utils\distutils_args.py
1:from distutils.errors import DistutilsArgError
2:from distutils.fancy_getopt import FancyGetopt

.\src\pip\_internal\operations\install\legacy.py
7:from distutils.util import change_root

.\src\pip\_internal\locations.py
14:from distutils import sysconfig as distutils_sysconfig
15:from distutils.command.install import SCHEME_KEYS  # type: ignore
16:from distutils.command.install import install as distutils_install_command
27:    from distutils.cmd import Command as DistutilsCommand
102:    from distutils.dist import Distribution

.\src\pip\_internal\cli\parser.py
12:from distutils.util import strtobool

.\src\pip\_internal\cli\cmdoptions.py
18:from distutils.util import strtobool

.\src\pip\_internal\build_env.py
9:from distutils.sysconfig import get_python_lib

The FancyGetopt usage should be replaced with standard option parsing (ideally the long-discussed "use a better option parser" rewrite). The strtobool usage should be trivial. locations.py should use sysconfig directly. That leaves install/legacy.py and build_env.py which are fairly isolated, but I haven't looked in detail at them.

[uranusjr]

I want to propose only do the changes on Python 3 since new setuptools is not available on Python 2 anyway. We can do this in the same framework as #8114. This way, the transition can start in 20.3, rather than 21.0 when Python 2 is dropped entirely.

PyCon Taiwan is already wrapping up (the main conference was held last weekend) and I plan to come back to the transition away from pkg_resources soon after that. The distutils.sysconfig usages fit nicely into the same process.

[McSinyx]

To add to the list, distutils.util.change_root (used by install.legacy) does not uses anything from distutils so it could be moved into pip just fine. Sometimes the function would be useful elsewhere too, e.g. to get the installation site if both root and target are specified (ref GH-7828).

Funny enough, distutils.sysconfig.get_python_lib is not available in sysconfig, but it can also be reimplemented just fine (it only uses an exception from distutils).

[uranusjr]

get_python_lib() is implemented as entries in sysconfig.get_paths(), pip only uses two of them:

• get_python_lib(plat_specific=True) → sysconfig.get_path('platlib')
• get_python_lib(plat_specific=False) → sysconfig.get_path('purelib')

(edit: I messed up the flags, the corresponding key is the other way around)

[pradyunsg]

@uranusjr do you wanna close this out, since you’re tracking the distutils migration in a different issue?

[uranusjr]

I think this can be closed.