Pip 19.1 pip install --no-use-pep517 not working with nested requirements files

[tom-dalton-fanduel]

Relevent issues:

#6375
#6314
#6370

Environment

• pip version: 19.1
• Python version: 3.6
• OS: Linux

Description

pip install fails with a requirements file that references a setup.py file, in a project that has a pyproject.toml file that has no build tooling section.

My project setup:

setup.py  # Contains the actual project/package requirements
pyproject.toml  # Contains only 2 lines of black configuration
requirements/base.txt  # Contains only `-e .`
requirements/testing.txt  # Contains `-r base.txt` and then other stuff like pytest etc.
src/mypackage/...  # Actual project/package code

This setup allows us to:

• Specify the project requirements exactly once (in the setup.py)
• Use a command like pip install -r requirements/testing.txt to install both the project requirements along with other stuff needed to run the tests.
• Build the package with python setup.py sdist

The suggested workaround in #6370 (comment) doesn't seem to work for me:

pip install --no-use-pep517 -q --upgrade --requirement /..../src/requirements/development.txt
ERROR: Error installing 'file:///.... (from -r /..../src/requirements/_base.txt (line 1))': editable mode is not supported for pyproject.toml-style projects. pip is processing this project as pyproject.toml-style because it has a pyproject.toml file. Since the project has a setup.py and the pyproject.toml has no "build-backend" key for the "build_system" value, you may pass --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP 517 for details on pyproject.toml-style projects.

pip install -r requirements/testing.txt fails

Expected behavior

Historically (in <= 19.0.3) pip would install the project and testing/dev requirements (into the current virtualenv).

How to Reproduce

Set up a project as per above, try to install the dev/testing requirements with pip install -r foo.txt.

Output

As per description.

[gaborbernat]

So I get now pip is PEP compliant, but this now breaks peoples workflow without any sane workaround, I consider it as a PEP oversight. PEP-517/8 on purpose did not want to cover editable installs, considering something to be addressed down the line. As such I would argue PEP-517/PEP-518 no longer applies when editable install mode is enabled. It's the build frontend dependent on how this case is handled for now. In such a case, pip should fallback to the old system, until we define editable installs inside a PEP. @pfmoore want to weigh in? If we want people to explicitly require the old behaviour that's okay, but then --no-use-pep517 should work independently if the pyproject.toml is there or what it contains.

[cjerdonek]

ERROR: Error installing 'file:///....

What does the "..." say? What is it trying to install at that point?

[pitrou]

Asking people to type the obscure --no-use-pep517 (should people even have to know what PEP 517 is?) when the well-known pip install -e used to work is not a sane workaround. I'm astonished that the pip maintainers estimate it's fine to break well-known practices just out of theoretical purity.

Of course, there's still python setup.py develop...

[con-f-use]

Is the underscore in _base.txt intendend. What happens when you specify --no-use-pep517 in the requirements file:

# .../requirements/_base.txt
--no-use-pep517 -e mypackage/
...

[cjerdonek]

pip install --no-use-pep517 -q --upgrade --requirement /..../src/requirements/development.txt

Also, your command invocation doesn't seem to match up with the project setup you listed because development.txt doesn't appear there. If you're nesting requirements files, it would be good to be clearer on the situation.

[tom-dalton-fanduel]

ERROR: Error installing 'file:///....

What does the "..." say? What is it trying to install at that point?

Sorry, I removed the paths for brevity. Here's the full output:

pip install --no-use-pep517 -q --upgrade --requirement /var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion/src/requirements/development.txt
ERROR: Error installing 'file:///var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion (from -r /var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion/src/requirements/_base.txt (line 1))': editable mode is not supported for pyproject.toml-style projects. pip is processing this project as pyproject.toml-style because it has a pyproject.toml file. Since the project has a setup.py and the pyproject.toml has no "build-backend" key for the "build_system" value, you may pass --no-use-pep517 to opt out of pyproject.toml-style processing. See PEP 517 for details on pyproject.toml-style projects.
make: *** [/var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion/_build/requirements.txt] Error 1

[tom-dalton-fanduel]

Sorry, we do nest requirements. I simplified this but the full hierarchy is:

requirements/development.txt : has -r testing.txt at the top, then dev-only requirements like ipython
requirements/testing.txt : has -r _base.txt at the top, then testing-only requirements like pytest
requirements/_base.txt : has one line -e .

[cjerdonek]

@tom-dalton-fanduel Have you tried adding the option inside the appropriate requirements file, instead of only at the root command?

@pitrou That's a fair point re: the name. I'm guessing the expectation was that the option would only have to be used infrequently. The only "user-friendly" name that PEP 517 provides for the new behavior is pyproject.toml-style (in contrast to setup.py-style). So maybe a friendlier alias would be something using those words (e.g. --setup-py-style or something along those lines).

[tom-dalton-fanduel]

@tom-dalton-fanduel Have you tried adding the option inside the appropriate requirements file, instead of only at the root command?

I had great hopes but it failed spectacularly!

Running virtualenv with interpreter /usr/bin/python3.6
pip 19.1 from /var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion/venv/lib/python3.6/site-packages/pip (python 3.6)
pip install -q --upgrade --requirement /var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion/src/requirements/development.txt
Usage: pip [options]

ERROR: Invalid requirement: --no-use-pep517 -e .
pip: error: no such option: --no-use-pep517

make: *** [/var/lib/jenkins/workspace/EC2_SQS_Queue_Deletion/sqs-deletion/_build/requirements.txt] Error 1

[pitrou]

Right, I wouldn't mind having to add a new option (though I'm not sure why -e couldn't imply it automatically), but --no-use-pep517 reminds me of --single-version-externally-managed, only worse :-)

[con-f-use]

@tom-dalton-fanduel : I seem to recall something about only certain pip options being allowed inside requirement files. Some where disallowed for security-related reasons.

In any case --no-use-pep517 should definitely be allowed there, and it's a separate issue if it isn't.

[Evpok]

I agree with @pitrou, given that PEP 517 does not define editable installs, asking for one should imply not using PEP 517.

[tom-dalton-fanduel]

Apologies for the confusion my failed attempts to simplify the problem. We do have a short term workaround in that we can just force-downgrade pip to 19.0.3. If we can't find a good workaround (or the intended workaround does not work in my case), I'd be keen to understand how we can effectively replicate our current setup.py-for-package and requirements-files-for-testing system using the new pyproject.toml file. It sounds like others are in the same boat so clear guidelines on best (or even good) practice for this would be helpful. At the moment I haven't been able to find a doc/blog/anything that explains what the migration path should/could be.