`_distutils_hack` fails if called anything but `get-pip.py`

[balupton]

Description

Pip fails to install:

which python
# /usr/local/opt/python/libexec/bin/python
/usr/local/opt/python/libexec/bin/python --version
# Python 3.9.10
down https://bootstrap.pypa.io/get-pip.py --destination=/var/folders/wm/8g56ry4s47z_hbt0tgv0m5gr0000gn/T/tmp.vv6opZCN
# down is an internal command
python /var/folders/wm/8g56ry4s47z_hbt0tgv0m5gr0000gn/T/tmp.vv6opZCN --user
# fails and outputs the next code block

/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py:17: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is in/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py:30File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/overrideFile "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py:17: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/var/folders/wm/8g56ry4s47z_hbt0tgv0m5gr0000gn/T/tmp.vv6opZCN", line 27081, in <module>
    main()
  File "/var/folders/wm/8g56ry4s47z_hbt0tgv0m5gr0000gn/T/tmp.vv6opZCN", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/var/folders/wm/8g56ry4s47z_hbt0tgv0m5gr0000gn/T/tmp.vv6opZCN", line 120, in bootstrap
    args = determine_pip_install_arguments()
  File "/var/folders/wm/8g56ry4s47z_hbt0tgv0m5gr0000gn/T/tmp.vv6opZCN", line 65, in determine_pip_install_arguments
    import setuptools  # noqa
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 71, in do_override
    ensure_local_distutils()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 59, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py

Expected behavior

it to work fine

pip version

latest

Python version

3.9.10

OS

21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64

How to Reproduce

this works fine:

brew install python
wget https://bootstrap.pypa.io/get-pip.py
chmod +x ./get-pip.py
python ./get-pip.py --user

this fails, which is common when say saving the installer to a randomly generated temp file:

brew install python
wget https://bootstrap.pypa.io/get-pip.py
mv ./get-pip.py ./any-other-filename.py
chmod +x ./any-other-filename.py
python ./any-other-filename.py --user

Output

Warning: python@3.9 3.9.10 is already installed and up-to-date.
To reinstall 3.9.10, run:
  brew reinstall python@3.9
--2022-01-19 23:48:26--  https://bootstrap.pypa.io/get-pip.py
Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 151.101.52.175, 2a04:4e42:d::175
Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|151.101.52.175|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2159352 (2.1M) [text/x-python]
Saving to: ‘get-pip.py’

get-pip.py                        100%[============================================================>]   2.06M  1.65MB/s    in 1.2s    

2022-01-19 23:48:28 (1.65 MB/s) - ‘get-pip.py’ saved [2159352/2159352]

/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py:17: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
  warnings.warn(
/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
  File "/Users/balupton/./any-other-filename.py", line 27081, in <module>
    main()
  File "/Users/balupton/./any-other-filename.py", line 139, in main
    bootstrap(tmpdir=tmpdir)
  File "/Users/balupton/./any-other-filename.py", line 120, in bootstrap
    args = determine_pip_install_arguments()
  File "/Users/balupton/./any-other-filename.py", line 65, in determine_pip_install_arguments
    import setuptools  # noqa
  File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 8, in <module>
    import _distutils_hack.override  # noqa: F401
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/override.py", line 1, in <module>
    __import__('_distutils_hack').do_override()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 71, in do_override
    ensure_local_distutils()
  File "/usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 59, in ensure_local_distutils
    assert '_distutils' in core.__file__, core.__file__
AssertionError: /usr/local/Cellar/python@3.9/3.9.10/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

Pinging @jaraco, since this comes out of the distutils hack that setuptools has.

[grv87]

Python 3.10.5, Windows 10, pip 22.1.2, setuptools 62.4.0 — the problem still exists.

I had an old version of setuptools installed, and this worked for me as a workaround:

1. First setup pip without setuptools:

python get-pip.py --no-setuptools

2. Then update setuptools:

pip install --upgrade setuptools

[pradyunsg]

This is realistically an issue with setuptools' hack, so I've moved the issue over.

[jaraco]

I'm sorry I missed the ping. I have hundreds of unread messages in my email inbox, so would have gotten to it eventually (most likely).

The distutils hack does have an exemption for pip to force it to use the stdlib distutils, but nothing specific to "get-pip.py", so I don't yet understand why the name of get-pip is relevant.

[jaraco]

Attempting to replicate the issue, I created this Dockerfile:

FROM ubuntu:jammy
RUN apt update
RUN apt upgrade -y
RUN apt install -y python3 python3-distutils wget
RUN wget https://bootstrap.pypa.io/get-pip.py -O othername.py
CMD python3 othername.py --user

Unfortunately, it works fine:

 draft $ docker run -it @$(docker build -q .)
Collecting pip
  Downloading pip-22.1.2-py3-none-any.whl (2.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 10.8 MB/s eta 0:00:00
Collecting setuptools
  Downloading setuptools-63.1.0-py3-none-any.whl (1.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 16.5 MB/s eta 0:00:00
Collecting wheel
  Downloading wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel, setuptools, pip
  WARNING: The script wheel is installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts pip, pip3 and pip3.10 are installed in '/root/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-22.1.2 setuptools-63.1.0 wheel-0.37.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Therefore, the issue appears to be restricted to some other factor in the user's environment (Python version, pip version, OS, Homebrew patches, ...).

Is it possible someone could create a Dockerfile that can replicate the issue?