PEP 518 support: setup.py is called before build requirements have been installed

[benoit-pierre]

• Pip version: 9.0.1-464-g573a5011
• Python version: Python 3.6.2
• Operating system: Arch Linux x86_64

Description:

I'd like to use pip' support for PEP 518, particularly to solve those 2 cases mentioned in the PEP:

• While setuptools itself will install anything listed in this, they won't be installed until during the execution of the setup() function, which means that the only way to actually use anything added here is through increasingly complex machinations that delay the import and usage of these modules until later on in the execution of the setup() function.
• This cannot include setuptools itself nor can it include a replacement to setuptools , which means that projects such as numpy.distutils are largely incapable of utilizing it and projects cannot take advantage of newer setuptools features until their users naturally upgrade the version of setuptools to a newer one.

For this to work I expect build requirements to be installed before any attempt at running setup.py.

What I've run:

> tmpdir="$(mktemp -d)"
> trap "cd - && rm -rf $tmpdir" EXIT
> cd "$tmpdir"
> python -m venv venv
> export PATH="$PWD/venv/bin:$PATH"
> pip install -U setuptools wheel https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip python-xlib
Collecting https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip
  Downloading https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip (6.5MB)
    100% |████████████████████████████████| 6.5MB 269kB/s
Collecting setuptools
  Using cached setuptools-36.2.7-py2.py3-none-any.whl
Collecting wheel
  Using cached wheel-0.29.0-py2.py3-none-any.whl
Collecting python-xlib
  Using cached python_xlib-0.19-py2.py3-none-any.whl
Collecting six>=1.10.0 (from python-xlib)
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: setuptools, wheel, six, python-xlib, pip
  Found existing installation: setuptools 28.8.0
    Uninstalling setuptools-28.8.0:
      Successfully uninstalled setuptools-28.8.0
  Found existing installation: pip 9.0.1
    Uninstalling pip-9.0.1:
      Successfully uninstalled pip-9.0.1
  Running setup.py install for pip ... done
Successfully installed pip-10.0.0.dev0 python-xlib-0.19 setuptools-36.2.7 six-1.10.0 wheel-0.29.0
> pip freeze --all
pip==10.0.0.dev0
python-xlib==0.19
setuptools==36.2.7
six==1.10.0
wheel==0.29.0
> cat >setup.py <<\EOF
from setuptools import setup
import Xlib
setup(name='project', version='0.1')
EOF
> cat >pyproject.toml <<\EOF
[build-system]
requires = ["python-xlib"]
EOF
> cat >MANIFEST.in <<\EOF
include pyproject.toml
EOF
> python setup.py sdist
running sdist
running egg_info
creating project.egg-info
writing project.egg-info/PKG-INFO
writing dependency_links to project.egg-info/dependency_links.txt
writing top-level names to project.egg-info/top_level.txt
writing manifest file 'project.egg-info/SOURCES.txt'
reading manifest file 'project.egg-info/SOURCES.txt'
writing manifest file 'project.egg-info/SOURCES.txt'
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

running check
warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

creating project-0.1
creating project-0.1/project.egg-info
copying files to project-0.1...
copying setup.py -> project-0.1
copying project.egg-info/PKG-INFO -> project-0.1/project.egg-info
copying project.egg-info/SOURCES.txt -> project-0.1/project.egg-info
copying project.egg-info/dependency_links.txt -> project-0.1/project.egg-info
copying project.egg-info/top_level.txt -> project-0.1/project.egg-info
Writing project-0.1/setup.cfg
creating dist
Creating tar archive
removing 'project-0.1' (and everything under it)
> pip uninstall -y python-xlib
Uninstalling python-xlib-0.19:
  Successfully uninstalled python-xlib-0.19
> pip install dist/project-0.1.tar.gz
Processing ./dist/project-0.1.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-c16oxefg/setup.py", line 2, in <module>
        import Xlib
    ModuleNotFoundError: No module named 'Xlib'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-req-build-c16oxefg/

[RonnyPfannschmidt]

pep518 is implemented in #4144 which will be released in pip 10, so currently there is no support in a released version

[pradyunsg]

You seem to be using the latest master. The issue you're facing (PEP 518 build requirements not being satisfied when installing from local sdists) would be fixed once a rebased #4562 merges.

On that note, I'll close this issue. :)

[benoit-pierre]

Correct me if I am wrong, but #4562 is about extending support to VCS link / directory installs. There's already support for installing from sdists. Case in point:

> tmpdir="$(mktemp -d)"
> trap "cd - && rm -rf $tmpdir" EXIT
> cd "$tmpdir"
> python -m venv venv
> export PATH="$PWD/venv/bin:$PATH"
> pip -q install -U setuptools wheel python-xlib https://github.com/pypa/pip/archive/573a50118148ff4646ae9310730f5bf6c4e8784f.zip
> pip freeze --all
pip==10.0.0.dev0
python-xlib==0.19
setuptools==36.2.7
six==1.10.0
wheel==0.29.0
> >setup.py <<\EOF
from setuptools import setup
from pkg_resources import working_set
print('in setup.py, available distributions:\n' +
      "\n".join(sorted(working_set.by_key.keys())))
setup(name='project', version='0.1')
EOF
> >pyproject.toml <<\EOF
[build-system]
requires = ["python-xlib"]
EOF
> >MANIFEST.in <<\EOF
include pyproject.toml
EOF
> python setup.py -q sdist
in setup.py, available distributions:
pip
python-xlib
setuptools
six
wheel
warning: sdist: standard file not found: should have one of README, README.rst, README.txt

warning: check: missing required meta-data: url

warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) must be supplied

> pip uninstall -y python-xlib
> pip install -v dist/project-0.1.tar.gz
Created temporary directory: /tmp/pip-install-eorc13vf
Processing ./dist/project-0.1.tar.gz
  Created temporary directory: /tmp/pip-req-build-054cpvg6
  Running command python -c "import setuptools"
  Running setup.py (path:/tmp/pip-req-build-054cpvg6/setup.py) egg_info for package from file:///tmp/tmp.TVomHQ96sm/dist/project-0.1.tar.gz
    Running command python -c "import setuptools"
    Running command python setup.py egg_info
    in setup.py, available distributions:
    pip
    project
    setuptools
    six
    wheel
    running egg_info
    creating pip-egg-info/project.egg-info
    writing pip-egg-info/project.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/project.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/project.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/project.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/project.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'pip-egg-info/project.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-054cpvg6 has version 0.1, which satisfies requirement project==0.1 from file:///tmp/tmp.TVomHQ96sm/dist/project-0.1.tar.gz
Building wheels for collected packages: project
  This version of pip does not implement PEP 516, so it cannot build a wheel without setuptools. You may need to upgrade to a newer version of pip.
  Created temporary directory: /tmp/pip-build-env-07bvfv7t
  1 location(s) to search for versions of python-xlib:
  * https://pypi.python.org/simple/python-xlib/
  Getting page https://pypi.python.org/simple/python-xlib/
  Looking up "https://pypi.python.org/simple/python-xlib/" in the cache
  Current age based on date: 441
  Freshness lifetime from max-age: 600
  Freshness lifetime from request max-age: 600
  The response is "fresh", returning cached response
  600 > 441
  Analyzing links from page https://pypi.python.org/simple/python-xlib/
    Found link https://pypi.python.org/packages/30/0a/ce07aa90f5086a643f70407769989162959147d14c871cba0e3822c8bc7a/python_xlib-0.19-py2.py3-none-any.whl#md5=eca5a5410a7090ab91263d11ac1340dc (from https://pypi.python.org/simple/python-xlib/), version: 0.19
    Found link https://pypi.python.org/packages/48/9a/bcf41728a0a81b3d76e2a873296c2912300c653f9e6453c760a50bd2ef93/python-xlib-0.19.tar.bz2#md5=81f26828fc4ccdcfae8e95de6409d129 (from https://pypi.python.org/simple/python-xlib/), version: 0.19
    Found link https://pypi.python.org/packages/5e/f6/1235775fe2a5cc0d788787b97b5911b208d2e093cc03d15a38175d53788b/python_xlib-0.17-py3-none-any.whl#md5=babd8a1dd7ded2ce08050dd9c50d3cf0 (from https://pypi.python.org/simple/python-xlib/), version: 0.17
    Skipping link https://pypi.python.org/packages/71/ca/0ecaa43b4403b9e56595190c6f2b93aae583fa189bb9acb61b57e75644c6/python_xlib-0.17-py2-none-any.whl#md5=c741c77e760abeb08873b70780fac3e9 (from https://pypi.python.org/simple/python-xlib/); it is not compatible with this Python
    Skipping link https://pypi.python.org/packages/bc/7a/ecdca237146993fbf988e78b0c7408d3b2bd8578496e9982c9cba2603e68/python_xlib-0.16-py2-none-any.whl#md5=9dd3f88d354536025161ccc902070770 (from https://pypi.python.org/simple/python-xlib/); it is not compatible with this Python
    Found link https://pypi.python.org/packages/d4/1b/a1ed5a5b47a6b4a1de029a157b61acf784e2fe8cf165fb6e32c627caa790/python-xlib-0.17.tar.bz2#md5=8b392bac1e634f605aa4dac3128dab95 (from https://pypi.python.org/simple/python-xlib/), version: 0.17
    Found link https://pypi.python.org/packages/d9/b3/189b3c786f41d416fddd90195b31d4b0031468bfd7b2392e5a3a9ccce393/python_xlib-0.18-py2.py3-none-any.whl#md5=d642624c44b0db855b611b1740bb8eb9 (from https://pypi.python.org/simple/python-xlib/), version: 0.18
    Found link https://pypi.python.org/packages/f6/11/a31b6ff171065f4e700914289036a1154dd8bfcb6b6e4c06a56c4052367f/python_xlib-0.16-py3-none-any.whl#md5=379107824aff037322b220fc444cf9ff (from https://pypi.python.org/simple/python-xlib/), version: 0.16
    Found link https://pypi.python.org/packages/fa/13/30ba501a3a92ceb35155798aa08f94cab15ca89a28deebcbbbc950c7c732/python-xlib-0.18.tar.bz2#md5=b54349fd0f3f6b9cbabf3a3303743758 (from https://pypi.python.org/simple/python-xlib/), version: 0.18
    Found link https://pypi.python.org/packages/ff/8e/0b63c8a086ca6bbbf0eafae1aa8e194ee63203c3f655f6e6c4ed24228646/python-xlib-0.16.tar.gz#md5=c60db93fd6d535d24a2bb4cc3e39e9f2 (from https://pypi.python.org/simple/python-xlib/), version: 0.16
  Using version 0.19 (newest of versions: 0.16, 0.17, 0.18, 0.19)
  Installing build dependencies ...   Running command /tmp/tmp.TVomHQ96sm/venv/bin/python -m pip install --ignore-installed --prefix /tmp/pip-build-env-07bvfv7t https://pypi.python.org/packages/30/0a/ce07aa90f5086a643f70407769989162959147d14c871cba0e3822c8bc7a/python_xlib-0.19-py2.py3-none-any.whl#md5=eca5a5410a7090ab91263d11ac1340dc
  Collecting python-xlib==0.19 from https://pypi.python.org/packages/30/0a/ce07aa90f5086a643f70407769989162959147d14c871cba0e3822c8bc7a/python_xlib-0.19-py2.py3-none-any.whl#md5=eca5a5410a7090ab91263d11ac1340dc
    Using cached python_xlib-0.19-py2.py3-none-any.whl
  Collecting six>=1.10.0 (from python-xlib==0.19)
    Using cached six-1.10.0-py2.py3-none-any.whl
  Installing collected packages: six, python-xlib
  Successfully installed python-xlib six
done
  Created temporary directory: /tmp/pip-wheel-us438joy
  Running command python -c "import setuptools"
  Running setup.py bdist_wheel for project ...   Destination directory: /tmp/pip-wheel-us438joy
  Running command /tmp/tmp.TVomHQ96sm/venv/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-054cpvg6/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-us438joy --python-tag cp36
  in setup.py, available distributions:
  pip
  project
  python-xlib
  setuptools
  six
  wheel
  running bdist_wheel
  running build
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_egg_info
  running egg_info
  writing project.egg-info/PKG-INFO
  writing dependency_links to project.egg-info/dependency_links.txt
  writing top-level names to project.egg-info/top_level.txt
  reading manifest file 'project.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'project.egg-info/SOURCES.txt'
  Copying project.egg-info to build/bdist.linux-x86_64/wheel/project-0.1-py3.6.egg-info
  running install_scripts
  creating build/bdist.linux-x86_64/wheel/project-0.1.dist-info/WHEEL
done
  Stored in directory: /home/bpierre/.cache/pip/wheels/2a/11/de/0c48087b5fbc5db1558414343195e81b473471967bde202c35
  Removing source in /tmp/pip-req-build-054cpvg6
Successfully built project
Installing collected packages: project

Successfully installed project-0.1
Cleaning up...

So build requirements are installed, it's just that setup.py has already been run once before...

I think my issue still stands.

[pradyunsg]

My bad - I guess I read something incorrectly. 😅

[benoit-pierre]

Thanks for reopening, don't forget to remove the invalid label too. ;)

[benoit-pierre]

Also note that my initial example was flawed, as pyproject.toml was not automatically included by setuptools in the sdist...

@benoit-pierre There is another subtle issue that I discovered while working on PEP 517 support. The existing "build environment" context manager doesn't load the entry_points from the temporary build environment. Does anyone know how pkg_resources's entry_points work, and more specifically, how to override the search location?

Ref: pypa/setuptools#1174.