New resolver downloads hundreds of different package versions, without giving reason

[anetbnd]

• pip version: 20.3.1
• Python version: Python 3.8.5
• Operating system: Windows 10 or Ubuntu

We switched today from pip 20.2.2 to 20.3.1 and suddenly in a big project with a very long dependency list (over 60 packages), pip does not manage anymore to install the dependencies from the requirements.txt file. It always tries to install hundreds of different package versions from the same package. Sometimes, it just tries out 3 or 10 versions, which takes some time but works in the end, but sometimes, it downloads and installs all possible versions it can find.

Here this is what happens with the package requests:

INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking 
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking 
INFO: pip is looking at multiple versions of requests to determine which version is compatible with other requirements. This could take a while. 
Collecting requests Downloading requests-2.24.0-py2.py3-none-any.whl (61 kB) 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl Downloading requests-2.23.0-py2.py3-none-any.whl (58 kB) 
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while. 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl Downloading requests-2.22.0-py2.py3-none-any.whl (57 kB) 
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl 
Downloading requests-2.21.0-py2.py3-none-any.whl (57 kB) 
Downloading requests-2.20.1-py2.py3-none-any.whl (57 kB) 
Downloading requests-2.20.0-py2.py3-none-any.whl (60 kB) 
Downloading requests-2.19.1-py2.py3-none-any.whl (91 kB) 
Downloading requests-2.19.0-py2.py3-none-any.whl (91 kB) 
Downloading requests-2.18.4-py2.py3-none-any.whl (88 kB) 
Downloading requests-2.18.3-py2.py3-none-any.whl (88 kB) 
Downloading requests-2.18.2-py2.py3-none-any.whl (88 kB) 
Downloading requests-2.18.1-py2.py3-none-any.whl (88 kB) 
Downloading requests-2.18.0-py2.py3-none-any.whl (563 kB) 
Downloading requests-2.17.3-py2.py3-none-any.whl (87 kB) 
Downloading requests-2.17.2-py2.py3-none-any.whl (87 kB) 
Downloading requests-2.17.1-py2.py3-none-any.whl (87 kB) 
Downloading requests-2.17.0-py2.py3-none-any.whl (87 kB) 
Downloading requests-2.16.5-py2.py3-none-any.whl (87 kB) 
Downloading requests-2.16.4-py2.py3-none-any.whl (87 kB) 
Downloading requests-2.16.3-py2.py3-none-any.whl (86 kB) 
Downloading requests-2.16.2-py2.py3-none-any.whl (86 kB) 
Downloading requests-2.16.1-py2.py3-none-any.whl (85 kB) 
Downloading requests-2.16.0-py2.py3-none-any.whl (85 kB) 
Downloading requests-2.15.1-py2.py3-none-any.whl (558 kB) 
Downloading requests-2.14.2-py2.py3-none-any.whl (560 kB) 
Downloading requests-2.14.1-py2.py3-none-any.whl (559 kB) 
Downloading requests-2.14.0-py2.py3-none-any.whl (559 kB) 
INFO: pip is looking at multiple versions of requests to determine which version is compatible with other requirements. This could take a while. 
Downloading requests-2.13.0-py2.py3-none-any.whl (584 kB) 
Downloading requests-2.12.5-py2.py3-none-any.whl (576 kB)
Downloading requests-2.12.4-py2.py3-none-any.whl (576 kB) 
Downloading requests-2.12.3-py2.py3-none-any.whl (575 kB) 
Downloading requests-2.12.2-py2.py3-none-any.whl (575 kB) 
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking 
Downloading requests-2.12.1-py2.py3-none-any.whl (574 kB) 
Downloading requests-2.12.0-py2.py3-none-any.whl (574 kB) 
Downloading requests-2.11.1-py2.py3-none-any.whl (514 kB) 
Downloading requests-2.11.0-py2.py3-none-any.whl (514 kB) 
Downloading requests-2.10.0-py2.py3-none-any.whl (506 kB) 
Downloading requests-2.9.2-py2.py3-none-any.whl (502 kB) 
Downloading requests-2.9.1-py2.py3-none-any.whl (501 kB) 
Downloading requests-2.9.0-py2.py3-none-any.whl (500 kB) 
Downloading requests-2.8.1-py2.py3-none-any.whl (497 kB) 
Downloading requests-2.8.0-py2.py3-none-any.whl (476 kB) 
Downloading requests-2.7.0-py2.py3-none-any.whl (470 kB) 
Downloading requests-2.6.2-py2.py3-none-any.whl (470 kB) 
Downloading requests-2.6.1-py2.py3-none-any.whl (469 kB) 
Downloading requests-2.6.0-py2.py3-none-any.whl (469 kB) 
Downloading requests-2.5.3-py2.py3-none-any.whl (468 kB) 
Downloading requests-2.5.2-py2.py3-none-any.whl (474 kB) 
Downloading requests-2.5.1-py2.py3-none-any.whl (464 kB) 
Downloading requests-2.5.0-py2.py3-none-any.whl (464 kB) 
Downloading requests-2.4.3-py2.py3-none-any.whl (459 kB) 
Downloading requests-2.4.2-py2.py3-none-any.whl (459 kB) 
Downloading requests-2.4.1-py2.py3-none-any.whl (458 kB) 
Downloading requests-2.4.0-py2.py3-none-any.whl (457 kB) 
Downloading requests-2.3.0-py2.py3-none-any.whl (452 kB) 
Downloading requests-2.2.1-py2.py3-none-any.whl (625 kB) 
Downloading requests-2.2.0-py2.py3-none-any.whl (623 kB) 
Downloading requests-2.1.0-py2.py3-none-any.whl (445 kB) 
Downloading requests-2.0.1-py2.py3-none-any.whl (439 kB) 
Downloading requests-2.0.0-py2.py3-none-any.whl (391 kB) 
Downloading requests-1.2.3.tar.gz (348 kB) 
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking 
Downloading requests-1.2.2.tar.gz (348 kB) 
Downloading requests-1.2.1.tar.gz (348 kB) 
Downloading requests-1.2.0.tar.gz (341 kB) 
Downloading requests-1.1.0.tar.gz (337 kB) 
Downloading requests-1.0.4.tar.gz (336 kB) 
Downloading requests-1.0.3.tar.gz (335 kB) 
Downloading requests-1.0.2.tar.gz (335 kB) 
Downloading requests-1.0.1.tar.gz (335 kB) 
Downloading requests-1.0.0.tar.gz (335 kB) 
Downloading requests-0.14.2.tar.gz (361 kB) 
ERROR: Command errored out with exit status 1: 
command: /opt/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-t94ak9d1 cwd: /tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/ 
Complete output (11 lines): 
Traceback (most recent call last): 
File "<string>", line 1, in <module> 
File "/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/setup.py", line 6, in <module> 
import requests 
File "/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/requests/__init__.py", line 52, in <module>
 from . import utils 
File "/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/requests/utils.py", line 22, in <module> 
from .compat import parse_http_list as _parse_list_header 
File "/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/requests/compat.py", line 112, in <module> 
from .packages import chardet2 as chardet 
ImportError: cannot import name 'chardet2' from 'requests.packages' (/tmp/pip-install-ilzlnu9j/requests_17e83d46f96d481997d94d444ebabe71/requests/packages/__init__.py) 
----------------------------------------

While I understand, that this is coming from the new resolver and a potential compatibility conflict with a certain package, I don't understand what is the exact issue here. We don't use requests directly, but it seems that this a dependency from a dependency we have. So we are not directly in control of this. And pip does not give me any help to understand, which packages have a version restriction, which lead to this behavior.

My suggestion is here to provide a better debug output for developers. When pip has to try out different version, it should tell why it must do so. For example:

Check package "requests-2.24.0-py2.py3-none-any.whl", but it can not be used, because package "<any package>" restricts the version to X and "<another package>" requests version Y. We try out an older version.
...

This would help me to understand where to begin with fixing those conflicts. At the moment I'm absolutely lost and have to switch back to pip 20.2.2.

[brainwane]

Hello and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us and including some thoughts on how to address it -- I agree that a more specific error message would be better. I am going to defer to the resolver developers on whether this is an issue they are already addressing in a different issue or whether it's distinct.

You probably saw, in that output, the line:

INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking

but maybe we should also link to more info there, or have it show up again later.

Just in case it's useful, I'll mention here some useful troubleshooting and workaround tips from the documentation:

• If pip is taking longer to install packages, read Dependency resolution backtracking for ways to reduce the time pip spends backtracking due to dependency conflicts.

• If you don’t want pip to actually resolve dependencies, use the --no-deps option. This is useful when you have a set of package versions that work together in reality, even though their metadata says that they conflict. For guidance on a long-term fix, read Fixing conflicting dependencies.

• In pip 20.3, you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

Do any of those tips help?

(If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the pip resolver beta period.)

[anetbnd]

Thank you very much! The information you provided is very helpful. I think I will use the legacy-resolver as long as I don't know the root cause for the issue.

Regarding your last question: I tested this also during the beta-phase and have seen the same issue. I reported the issue over https://pip.pypa.io/surveys/backtracking

But it is also my fault, I did not spend so much time in understanding the root cause then. I was thinking that this is an code error in pip rather than a dependency error, because I have not seen a clear dependency conflict statement and could not believe that this is a "normal" behavior in such cases. Thus I just filled the survey and forgot it until today.

[anetbnd]

One further comment from my side: When I read the documentation the overall statement is "be more strict" to avoid those issues above.
While it is clear, that a higher restriction in choosing the libraries are reducing the search space and can in a individual case help to reduce the pip-runtime, I think in general this is a very bad advice. Because when developer starts to be more strict with the library version, also those projects which are libraries, them we will get more compatibility issues with pip in future.

Wouldn't it be better to say "be as open as possible and as strict as necessary" in order to guarantee, that most libraries will also in future work together?

[lelit]

I just hit the same: I have the habit of being as strict as possible in my constraints.txt files, pinning almost every needed package, and their dependants, recursively. I quickly discovered what was going on, and was able to fix the issue because luckily it was in one of my own packages, but I worry about possible similar cases where the fix won't be so easy.
In my case, the faulty package declared a dependency on celery[redis], and, even if the constraints.txt contains celery==4.4.7 and redis==3.5.3, the resolver wasn't able to handle the case, backtracking thru all celery versions, starting from current 5.0.3 going back until 3.0.0, where it stopped because that version's setup.py failed with an error.
The fix was simple: I splitted the celery[redis] dependency in two lines, i.e. I made my package depends on celery and redis.
Given that one cannot put celery[redis]==4.4.7 in the constraints.txt, I guess that's the only option. But it is easy to imagine how difficult it can be to disentangle the tree when packages out of my control declare dependencies using extras...
Anyway, couldn't the resolver at least consider the constraints I provide, and avoid looking for versions of celery that for sure won't be accepted? I mean, if I tell that I want celery==4.4.7, what's the point of downloading all its historic versions and even higher versions?

[pradyunsg]

@lelit could you make a Github Gist with reproduction instructions for this? The resolver should be considering the constraints you pass it and restrict the versions it explores based on it.

[lelit]

Sure, I'll try to distill a recipe.

[lelit]

Ok, got it!
As the comment in constraints.txt says, the problem is triggered by the presence of an unsolvable dependency.
See https://gist.github.com/lelit/50410bec3e31bd98af8a1109fed2d673: executing pip -c constraints.txt -r requirements.txt results in

$ pip install -c constraints.txt -r requirements.txt 
Collecting celery[redis]
  Using cached celery-5.0.3-py3-none-any.whl (392 kB)
  Using cached celery-5.0.2-py3-none-any.whl (392 kB)
  Using cached celery-5.0.1-py3-none-any.whl (392 kB)
  Using cached celery-5.0.0-py3-none-any.whl (389 kB)
  Using cached celery-4.4.7-py2.py3-none-any.whl (427 kB)
Collecting pgcli
  Downloading pgcli-3.0.0-py3-none-any.whl (70 kB)
     |████████████████████████████████| 70 kB 719 kB/s 
INFO: pip is looking at multiple versions of celery[redis] to determine which version is compatible with other requirements. This could take a while.
Collecting celery[redis]
  Using cached celery-4.4.6-py2.py3-none-any.whl (426 kB)
  Using cached celery-4.4.5-py2.py3-none-any.whl (426 kB)
  Using cached celery-4.4.4-py2.py3-none-any.whl (426 kB)
  Using cached celery-4.4.3-py2.py3-none-any.whl (424 kB)
  Using cached celery-4.4.2-py2.py3-none-any.whl (422 kB)
  Using cached celery-4.4.1-py2.py3-none-any.whl (422 kB)
  Using cached celery-4.4.0-py2.py3-none-any.whl (421 kB)
  Using cached celery-4.3.1-py2.py3-none-any.whl (415 kB)
....

[pradyunsg]

Hurray! Thanks for sharing. I'm AFK at the moment, but if you could you try that with the current master branch, that'd be awesome. There's one fairly major fix merged since the release that'd make things more... efficient.

It can be installed with pip install git+https://github.com/pypa/pip.git.

[lelit]

@pradyunsg: using current master the "backtrace" moves to another package (pytz), while for celery it still visits “incompatible” v5.0.x versions.

[hwalinga]

I had a related problem, but it was somewhat worse, because seemingly it not only downloads them, but also verifies them. And there was one version of this package from 4 years ago that was mispackaged, and pip immediately aborted mission. See spyder-ide/spyder#14365

And there is more, because after introspecting the dependency tree of Spyder, all dependencies requiring the package decorator don't specify any version constraint. Pip should just immediately install the newest version.

Based on this I have a few recommendations:

• When backtracking don't completely abort mission when a package is mispackaged.
• Don't initialize backtracking when none of the dependents specify any maximum version, just install the latest version
• Don't download and introspect all possible versions when backtracking, but evaluate them from most recent to oldest and immediately install the first version that satisfies the requirements.