pip download should handle ABI3 tags better

[dan-blanchard]

What's the problem this feature will solve?
I'm trying to download all of the manylinux wheels for the packages in our requirements.txt file by running:

pip download --python-version 36 --platform manylinux1_x86_64 --implementation cp --abi cp36m --only-binary=:all: -r requirements.txt

It works very well until it encounters a package for which the ABI tag is abi3, like Cryptography's latest releases use.

Describe the solution you'd like

I would like to be able to run pip download once and have it download all of the wheels compatible with the specified version-platform-implementation. pip download should be smart enough to know abi3 is compatible with the same interpreter that has the cp36m ABI tag and that python-version doesn't matter in the ABI3 case either.

Alternative Solutions

I could manually split up the requirements file into ones with the abi3 tag and ones with cp36m tag, but I have no way of knowing which one will be which in advance, and they could change in the future.

Additional context
I really only want this to work so that I can use shiv cross-platform and it offloads all the package downloading duties to pip download.

[zooba]

I just did some testing of this (from the issue linked from Azure Functions) and it seems that even specifying --abi abi3 isn't enough to get the cryptography wheels unless you also set --python-version 3.4.

I proposed a workaround at Azure/azure-functions-core-tools#1150 which should make everyone sad enough to think about fixing this :)

There's also pyca/cryptography#4382 for the cryptography wheel in particular, though it's already been closed (pip install works fine, but not pip download with an alternate platform specified). I pinged to try and resurrect, so we'll see whether they fix their own wheel first, but I think this is an inconsistency that ought to be resolved in pip as well.

Any thoughts on the best way of dealing with this? I can probably rustle up some dev time to work on it if we have an idea of the scale of the effort.

[reaperhulk]

It definitely seems like pip should be using the same tag compatibility resolution paths when downloading for alternate platforms as it does normally. Then everything would work as expected here. @njsmith mentioned Brett was doing some work to rewrite tag handling so maybe that work can apply here.

[njsmith]

Currently, tags <-> systems is a bipartite graph between two different types of things. You can go from one to the other: a tag tells you which kinds of systems it can run on, and given a running environment we can sniff out which tags it can support (by calling various functions, digging around in various modules, etc.). But we don't have any vocabulary for naming a system, or any way to expand from a single tag to the set of compatible tags.

@brettcannon has been working on a new tag handling library, and IIRC one of the things he's been trying to do is to give systems unique names and map from those names to the set of corresponding tags... I'm not sure how successful that's been, but I think it's the closest thing so far to what Azure needs here.

[brettcannon]

The work for unifying the tag code in a single package is sitting in a PR at pypa/packaging#156 waiting for a review.

[njsmith]

@brettcannon Did you end up with a system where you go

environment -> single tag describing the environment -> all the tags compatible with that environment

? IIUC that's what we need to support this Azure use case...

[brettcannon]

@njsmith not exposed to users, but the code should be able to do that. Currently it goes:

query current environment -> all tags compatible with the current environment

Stick your proposal in the middle of that should just be tweaking the API and some of the flow of the code (especially since I tried to make the code testable so you can pass in specific things like an ABI to get back the list of all other supported ABIs, it just might need some tweaks around current assumptions).

[chrahunt]

This should be fixed by #7354 since packaging.tags adds "abi3" for CPython even when an abi is explicitly provided and backfills in older versions (which pip doesn't currently do).

[pradyunsg]

I'll let someone chime in, in case ab3 isn't working for them. If it is, let us know. :)

I'll add a label to auto-close this issue if no one responds.