ERROR: No matching distribution found for torch==1.11.0+cu113 when using --extra-index-url

[DanielPerezJensen]

It seems that this error is also a problem when installing from other sources, such as in #5021.

Issue description

After lots of debugging and trial and error, we have run into a bug with pipenv relating to installing libraries while using another index url (--extra-index-url).

When trying to install PyTorch using the following command pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113" Installing torch==1.11.0+cu113, as specified under this issue: #4961, we get the following error: ERROR: No matching distribution found for torch==1.11.0+cu113.

According to @Bananaman this was possible to do in a previous version of pipenv.

Expected result

Expected an installation using a 3rd party index url, in this case PyTorch.

Actual result

daniel@daniel-desktop:~/Documents$ pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"
Installing torch==1.11.0+cu113...
Adding torch to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock (db4242) out of date, updating to (7d86a3)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 

CRITICAL:pipenv.patched.notpip._internal.resolution.resolvelib.factory:Could not find a version that satisfies the requirement torch==1.11.0+cu113 (from versions: 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 743, in _main
[ResolutionFailure]:       resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 704, in resolve_packages
[ResolutionFailure]:       results, resolver = resolve(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/resolver.py", line 685, in resolve
[ResolutionFailure]:       return resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1398, in resolve_deps
[ResolutionFailure]:       results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 1127, in actually_resolve_deps
[ResolutionFailure]:       resolver.resolve()
[ResolutionFailure]:   File "/home/daniel/.local/lib/python3.8/site-packages/pipenv/utils.py", line 905, in resolve
[ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: No matching distribution found for torch==1.11.0+cu113

Steps to replicate

Be sure to use the latest version of pipenv, if I use a previous version (latest confirmed is 2022.1.8) the error goes away.

1. Create clean environment
2. pipenv install --extra-index-url https://download.pytorch.org/whl/cu113/ "torch==1.11.0+cu113"
3. Error should appear on latest version of pipenv

---

$ pipenv --support

Pipenv version: '2022.3.28'

Pipenv location: '/home/daniel/.local/lib/python3.8/site-packages/pipenv'

Python location: '/usr/bin/python3'

Python installations found:

• 3.8.12: /home/daniel/.pyenv/versions/3.8.12/bin/python
• 3.8.10: /usr/bin/python3
• 3.8.10: /usr/bin/python3.8
• 3.8.10: /bin/python3
• 3.8.10: /bin/python3.8

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.10',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.13.0-37-generic',
 'platform_system': 'Linux',
 'platform_version': '#42~20.04.1-Ubuntu SMP Tue Mar 15 15:44:28 UTC 2022',
 'python_full_version': '3.8.10',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• SESSION_MANAGER
• QT_ACCESSIBILITY
• COLORTERM
• PYENV_SHELL
• XDG_CONFIG_DIRS
• XDG_MENU_PREFIX
• GNOME_DESKTOP_SESSION_ID
• MANDATORY_PATH
• LC_ADDRESS
• GNOME_SHELL_SESSION_MODE
• LC_NAME
• SSH_AUTH_SOCK
• XMODIFIERS
• DESKTOP_SESSION
• LC_MONETARY
• SSH_AGENT_PID
• GTK_MODULES
• PWD
• LOGNAME
• XDG_SESSION_DESKTOP
• XDG_SESSION_TYPE
• GPG_AGENT_INFO
• XAUTHORITY
• GJS_DEBUG_TOPICS
• WINDOWPATH
• HOME
• USERNAME
• IM_CONFIG_PHASE
• LC_PAPER
• LANG
• LS_COLORS
• XDG_CURRENT_DESKTOP
• VTE_VERSION
• GNOME_TERMINAL_SCREEN
• INVOCATION_ID
• MANAGERPID
• GJS_DEBUG_OUTPUT
• LESSCLOSE
• XDG_SESSION_CLASS
• SETUPTOOLS_USE_DISTUTILS
• TERM
• LC_IDENTIFICATION
• DEFAULTS_PATH
• LESSOPEN
• USER
• GNOME_TERMINAL_SERVICE
• DISPLAY
• SHLVL
• LC_TELEPHONE
• QT_IM_MODULE
• LC_MEASUREMENT
• LD_LIBRARY_PATH
• XDG_RUNTIME_DIR
• PYENV_ROOT
• LC_TIME
• JOURNAL_STREAM
• XDG_DATA_DIRS
• PATH
• GDMSESSION
• DBUS_SESSION_BUS_ADDRESS
• LC_NUMERIC
• OLDPWD
• _
• PIP_SHIMS_BASE_MODULE
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/daniel/.pyenv/shims:/home/daniel/.pyenv/bin:/usr/local/cuda-11.3/bin:/home/daniel/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/daniel/Documents

---

[Arcitec]

Excellent, thanks for creating the ticket. I have managed to verify that 2022.1.8 version of Pipenv works:

pip install --user pipenv==2022.1.8

I have not checked all of the March 2022 releases but I know for a fact that Pipenv 2022.3.28 is broken.

So the bug was introduced somewhere in the March releases.

[matteius]

Please seem my comment here, I suspect this is an edge case of the fix for package index restrictions. If you name the index and specify you want it from the named index it should work on newer versions. Can you verify that point?
#5021 (comment)

What do you expect the Pipfile to look like when you are using the --extra-index-url argument (try on the older 2022.1.8 release)?

[matteius]

I was looking in the pipenv docs for where it talks about --extra-index-url and not finding it. The index examples are generally named, or pypi mirrors. Ref: https://pipenv.pypa.io/en/latest/advanced/#specifying-package-indexes

I do see in the code base we have that option, and the help text is:
URLs to the extra PyPI compatible indexes to query for package look-ups.

Likely a regression that we never had a test case for, but more thought will need to go into it, especially considering the security concerns of package indexes.

I have a hunch that for your use case you really should be restricting the package index by name, otherwise pip will still go look in pypi and potentially pick something from there if the version is newer. If you truly want to ensure you are using the specific index that hosts your torch distribution, I would pin to that.

[DanielPerezJensen]

@matteius

Under the older release my Pipfile looks like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu113/"
verify_ssl = true
name = "downloadpytorch"

[packages]
torch = "==1.11.0+cu113"
numpy = "*"

[dev-packages]

[requires]
python_version = "3.8"

How do I name the index?

[matteius]

@DanielPerezJensen It would be like this:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu113/"
verify_ssl = true
name = "downloadpytorch"

[packages]
torch = {index="downloadpytorch", version="==1.11.0+cu113"}
numpy = "*"

[dev-packages]

[requires]
python_version = "3.8"

[DanielPerezJensen]

@matteius Would I have to specify that in the Pipfile then by hand or is there a way to get it written like that through terminal commands?

[Arcitec]

@DanielPerezJensen It seems very likely that the breaking change was the "stricter index searching" which @matteius mentioned.

Indeed, the non-CUDA versions of Torch are listed as candidates in the error message, which suggests that Pipenv is searching on PyPi for Torch and is ignoring the Torch index.

The Pipfile now needs a "index=XYZ" line to explicitly say "get torch from the downloadpytorch index/source".

It's a bug/regression in Pipenv which makes it ignore the contents of the specified extra sources even though we want them.

And @matteius: The --extra-index-url is just for adding another source. It doesn't seem to create a package pin/lock to that repo. Which is fine in Torch's cases since the versions we want only exist on the Torch repo/index.

[Arcitec]

@matteius I can confirm that manually editing the Pipfile to force Pipenv to get the package from PyTorch's repo works as a temporary solution.

Any ideas if the general bug can be fixed so that Pipenv scans the additional repos again without having to be forcibly told to use them per-package?

Related ticket: #4983

A fix may be something like:

• If "index" is specified on a Pipfile's package line, use that index.
• If no index is specified, prefer the non-PyPi sources (because the user added extra sources for a reason!), then lastly PyPi as fallback.

[matteius]

The --extra-index-url is just for adding another source. It doesn't seem to create a package pin/lock to that repo. Which is fine in Torch's cases since the versions we want only exist on the Torch repo/index.

Understood, but I know from triaging your other reports that it does in fact check pypi when you are using the extra index url, and that seems dangerous and takes additional time to do the locking and install than just being explicit about where you know that you want the package from.

Any ideas if the general bug can be fixed so that Pipenv scans the additional repos again without having to be forcibly told to use them per-package?

I will be studying the code more to consider your use case, will have to consider what might be possible before getting back to you on the solution wrt to that.

If no index is specified, prefer the non-PyPi sources, then lastly PyPi as fallback.

This part will likely not be possible because this is a pip behavior and we try to use pip in a way that does not modify its internals wherever possible and that would be quite the modification to do -- in fact pip prefers PyPi and says its recommended that if you want to avoid pypi but need pypi packages that your mirror should contain all pypi packages that you need (or mirror pypi). We might be able to get back the old behavior if no index is specified though.

[Arcitec]

Understood, but I know from triaging your other reports that it does in fact check pypi when you are using the extra index url, and that seems dangerous and takes additional time to do the locking and install than just being explicit about where you know that you want the package from.

Is there any way to modify --extra-index-url so that it means "add this index (if missing) AND force the packages (on the same command line) to use this index" then? Otherwise --extra-index-url is totally useless now since packages aren't being read from it.

May wanna remove --extra-index-url and add --use-index-url or --set-index-url though, to be explicit about that flag's new behavior.

This part will likely not be possible because this is a pip behavior and we try to use pip in a way that does not modify its internals wherever possible and that would be quite the modification to do -- in fact pip prefers PyPi and says its recommended that if you want to avoid pypi but need pypi packages that your mirror should contain all pypi packages that you need (or mirror pypi). We might be able to get back the old behavior if no index is specified though.

I see. So you're internally calling the pip program and telling it to install from the pip/index?

I guess all of this is fixed/good enough if --extra-index-url is modified to pin all packages on the same command line to use that index.

It's all a bit weird now though. It used to be possible to do something like pipenv install --extra-index-url url1 --extra-index-url url2 somepacgefrompypi somepackagefromurl1 somepackagefromurl1 somepackagefromurl2 to do it all at once, since Pipenv was able to look for the package name and version in all extra repos in addition to checking PyPi.

That behavior was the best. If someone has a reason to manually pin a package to an exact index in the Pipfile then so be it. But for general easy usage, "pick the matching package and version from any repo that has it" was great.

[matteius]

Is there any way to modify --extra-index-url so that it means "add this index (if missing) AND force the packages (on the same command line) to use this index" then? Otherwise --extra-index-url is totally useless now since packages aren't being read from it.

pipenv install "torch==1.11.0+cu113" --index https://download.pytorch.org/whl/cu113/ -v
This yields pip command:
python -m pip install --verbose --upgrade --exists-action=i -r 'c:\users\matte\appdata\local\temp\pipenv-v6p5_r5y-requi rements\pipenv-gwi7sw6w-requirement.txt' -i https://download.pytorch.org/whl/cu113/ --extra-index-url https://pypi.org/simple --extra-index-url https://pypi.org/simple

It's still running because its a huge package with multiple matching wheels, so debugging this will take some time.

May wanna remove --extra-index-url and add --use-index-url or --set-index-url though, to be explicit about that flag's new behavior.

Going to give it more thought, ideally we get the prior behavior working again for you folks that don't use index restrictions, but that is going to require some thinking time.

I see. So you're internally calling the pip program and telling it to install from the pip/index?

Yes, and more complicated than that is we vendor (copy in a specific version of pip) to the pipenv code base, and we call internal APIs of pip to do magical things that even pip itself doesn't do. For more insights into that, I have a recent thread in pip that goes into some detail around some of the complexities: pypa/pip#10964 (comment)

It's all a bit weird now though. It used to be possible to do something like pipenv install --extra-index-url url1 --extra-index-url url2 somepacgefrompypi somepackagefromurl1 somepackagefromurl1 somepackagefromurl2 to do it all at once, since Pipenv was able to look for the package name and version in all extra repos in addition to checking PyPi.

Yeah, I think that is why it will be necessary to contemplate getting that prior behavior working again without breaking index restrictions. I have to note, when calling out that many extra index urls, its still going to prefer pypi and its going to cause more churn on the resolver/locking to find your packages.

That behavior was the best.

Debatable :-) It is also great for package confusion attackers because how do you know for sure where you are pulling your package from in that scenario?

"pick the matching package and version from any repo that has it"

So imagine someone can poison the DNS for one of the package servers and put up a private pypi with a version that is malicious -- now your resolver goes and gets that because it matches version="==1.11.0+cu113" even though its not at all the version you wanted.

I'm glad to have having this discussion and thankful you opened the reports, so it will get investigated. For now you have two workaround, the prior version or being more explicit in your package needs which will I think benefit you more in the long run. For example, @Bananaman did you ever consider my other comment about calling out the wheel you specifically want to avoid downloading all 8 and having them in the lockfile? It really depends on if you need the lockfile to have all 8 hashes for the different system architectures or if you are just trying to support the one for your system. Ref: #4963 (comment)

[DanielPerezJensen]

The installing of specific wheels also serves well for repo's that might not conform to PEP503. For example, PyTorch Geometric

[matteius]

@matteius Would I have to specify that in the Pipfile then by hand or is there a way to get it written like that through terminal commands?

@DanielPerezJensen Yes it seems that this works: pipenv install "torch==1.11.0+cu113" --index https://download.pytorch.org/whl/

Generated this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu113/"
verify_ssl = true
name = "downloadpytorch"

[[source]]
url = "https://download.pytorch.org/whl/"
verify_ssl = true
name = "downloadpytorch-390"

[packages]
torch = {version = "==1.11.0+cu113", index = "downloadpytorch-390"}

[dev-packages]

[requires]
python_version = "3.10"

Initial reaction here is I am kind of surprised it has 3 sources, but it seemed to have worked. Generated this Pipfile.lock ... actually this doesn't look right either because it doesn't have the package hashes in the lockfile.

{
    "_meta": {
        "hash": {
            "sha256": "dd84b90e2afa892488dc59130dc48898573afeee5ba3aa45d8c64c54be6be39c"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "downloadpytorch",
                "url": "https://download.pytorch.org/whl/cu113/",
                "verify_ssl": true
            },
            {
                "name": "downloadpytorch-390",
                "url": "https://download.pytorch.org/whl/",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "torch": {
            "index": "downloadpytorch-390",
            "version": "==1.11.0+cu113"
        },
        "typing-extensions": {
            "hashes": [
                "sha256:1a9462dcc3347a79b1f1c0271fbe79e844580bb598bafa1ed208b94da3cdcd42",
                "sha256:21c85e0fe4b9a155d0799430b0ad741cdce7e359660ccbd8b530613e8df88ce2"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==4.1.1"
        }
    },
    "develop": {}
}

Side-note: I was having some buggy experiences earlier using git shell in windows to the locking, and I can only figure the size of the wheel files were freezing up the shell. I just had a much much better experience using Windows Powershell, it was very fast actually. I wonder if the reason its so fast is because its not pulling the hash values anymore.