setuptools 50 breaks pip installation on Debian/Ubuntu

[JelleZijlstra]

On newly created Ubuntu 16.04 AWS instances, we started seeing that python3.6 -m pip no longer works after get-pip.py (version 19.3.1). get-pip puts pip in /usr/lib/python3.6/site-packages/pip-20.2.2.dist-info, but sys.path for our 3.6 executable is ['', '/usr/lib/python36.zip', '/usr/lib/python3.6', '/usr/lib/python3.6/lib-dynload', '/usr/local/lib/python3.6/dist-packages', '/usr/lib/python3/dist-packages'].

On a newly created box I was able to confirm that using SETUPTOOLS_USE_DISTUTILS=stdlib fixes the problem:

    1  python3.6 /get_pip.py 
    2  python3.6 -m pip  # works
    3  python3.6 -m pip install -U setuptools  # updates from 41.0.1 to 50.0.0
    4  python3.6 -m pip  # still works
    5  python3.6 /get_pip.py 
    6  python3.6 -m pip  # can't find pip any more
    7  SETUPTOOLS_USE_DISTUTILS=stdlib python3.6 /get_pip.py 
    8  python3.6 -m pip  # works again

Sorry this isn't a completely self-contained repro case for the bug; hopefully this is enough information to figure out what's going on.

[ianw]

What you're seeing is pypa/setuptools#2232 which has been re-introduced, mentioned pypa/setuptools@04e3df2

In short, packaged Debian/ubuntu Python installs things into dist-packages. site-packages is left alone in-case you like to build your own upstream python and install it. Debuntu overrides the distutils it ships to redirect installs to dist-packages.
It gets blurry when you install pip from upstream. But it would call setuptools, which would use the patched distutils so things worked.

Now setuptools vendors distutils this and always installs things into site-packages, which isn't on the packaged python interpreter path, so it can't find modules.

Another consequence is the binary location, which moved from /usr/local/bin/ to /usr/bin. That's kind of a moot point though, because your binaries won't be able to find their libraries anyway.

[vmarkovtsev]

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

[ianw]

Another consequence of this is broken development installations on Ubuntu (pip install -e). pth files are written in site-packages instead of dist-packages and thus do not work.

I think it's all related; if you hit the inbuilt distutils you're not getting the overrides that Debian has set. So pip install -e is the most straight forward way to do this.

It is confusing however; if you just do pip install . then you'll build and install a wheel, which won't exhibit this problem. But if you try pip install --no-binary :all: . in your source directory it will be using distutils and and install it to site-packages, not dist-packages. You could also run setup.py directly which will exhibit the problem too.

[pradyunsg]

Quoting the changelog entry, for the relevant change:

pypa/setuptools#2232: Once again, Setuptools overrides the stdlib distutils on import. For environments or invocations where this behavior is undesirable, users are provided with a temporary escape hatch. If the environment variable SETUPTOOLS_USE_DISTUTILS is set to stdlib, Setuptools will fall back to the legacy behavior. Use of this escape hatch is discouraged, but it is provided to ease the transition while proper fixes for edge cases can be addressed.

Please see pypa/setuptools#2232 for the relevant discussion on this.

[encukou]

Note that this affects other distros than just Ubuntu/Debian. It seems that distutils is the only place where you can define install schemes (and even if that's not the case, people assume that). So, it's quite widely patched.
See also: https://discuss.python.org/t/how-can-we-keep-pip-from-updating-uninstalling-system-libraries-in-usr-lib-python/5059