Multi Platform support for Pipfile.lock

Is your feature request related to a problem? Please describe.

Currently it is not possible to create a Pipfile.lock for multiple platforms automatically.
Only hashes for the current platform are locked.
This is problematic when you want to use Pipefile.lock to share locked dependency versions in a multi platform setup, e.g. development on Windows or Mac and production on Linux.

E.g. cryptography generates the following Pipfile.lock entry on Windows 10 64bit:

        "cryptography": {
            "hashes": [
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==37.0.2"
        },

and the following one on a Linux VM:

        "cryptography": {
            "hashes": [
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e"
            ],
            "markers": "python_version >= '3.6'",
            "version": "==37.0.2"
        },

Describe the solution you'd like

My preferred solution would be a new entry in the Pipfile that defines which platforms hashes must always be included when locking.
If the current systems platform isn't in the list it should be be added to the lock list though to retain backwards compatibility.

The Pipfile entry could look like this:

[[source]]
platforms = "manylinux_2_24_x86_64 or win_amd_64"

If possible it would be great if the values could be eagerly validated and an error message printed on invalid values.

Note: Ticket #210 is similar to this one, but I was asked to open a new ticket

[dijital20]

Could the lock file just be per-platform/version/etc?

Pipfile defines my sources and packages.

• When I pipenv lock on my Python 3.8 on Windows 64bit, why couldn't it produce a Pipfile.3.8.windows.x64.lock?
• If I try to pipenv sync on Python 3.10, since there is not Pipfile.3.10.windows.x64.lock, could the normal behavior kick in and a new lock would be generated?
• Likewise, if I pipenv lock on Linux python 3.10, could it generate a Pipfile.3.10.linux.x64.lock?

What would be the drawback to that type of approach, aside from legacy support for the existing Pipfile.lock?

[matteius]

What would the advantage of maintaining multiple lock files per platform and environment @dijital20 ? I am not a big fan of the idea of having platform specific lock files and then how to know which one to use during sync.

could the normal behavior kick in and a new lock would be generated?

Not for sync -- that would be very very bad because you loose the trust that you are verifying the dependencies you used in dev/test are the same you are deploying to production because the hashes would regenerate.

The current design can be extended to account for multiple platforms already -- for multi-platform dependencies all of the associated hashes are included in the lock, so this issue is really about dependencies that are for a different platform than the one you are locking on should also be included in this lock file with the relevant platform marker. Today this only happens if you lock on the environment that requirement is compatible with. Pipenv's own lock file has this problem -- if we do not lock on windows we loose a windows-specific dependency from the lock file.

EDIT: Just re-read the issue report and I am very surprised that the list of hashes for cryptography would be different between those two environments -- this part deserves some additional triage with the latest version of pipenv.

[dijital20]

Yeah, platform-specific and Python-version-specific dependencies are the issue to solve here.

I had similar issues though when I moved a project from Python 3.8 to Python 3.10. My Pipfile and lock files both had Python 3.8 in them, and the standard pipenv sync --dev failed unless I did pipenv sync --dev --python C:\python310\python.exe, and then re-locked. At that point, everyone using Python 3.8 was out of luck, and so was anyone on Python 3.10.

I am not a big fan of the idea of having platform specific lock files and then how to know which one to use during sync.

pipenv when called, would generate the lock file name based on the interpreter. In my examples, the names were Pipfile.{version}.{platform}.{architecture}.lock. If the file exists, party on. If the file doesn't exist, then it should behave the same way pipenv does today if the lock file doesn't exist.

I am not sure if platform-specific lock files is the solution... just thinking through it as we ran into this today, as a colleague is developing a new Python module that he wants to support both Windows and Linux on, but there are some dependencies which are platform specific.

Some alternatives to consider:

• Build the multi platform stuff into the lock file itself, but ideally in a better way than it is now.
• Add an option to lock, sync, install, and uninstall to specify a lock file name/path. If not specified, use the default name today. This gives the user option to maintain separate specific lock files for different environments.

[matteius]

@dijital20 or @xyxz-web -- were either of you by chance using private python repositories isntead of pypi? I just locked on the latest pipenv main on windows and cryptography has a lot more hashes -- like all of them -- than the original example had.

        "cryptography": {
            "hashes": [
                "sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
                "sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
                "sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
                "sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
                "sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
                "sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
                "sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
                "sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
                "sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
                "sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
                "sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
                "sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
                "sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
                "sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
            ],
            "index": "pypi",
            "version": "==37.0.2"
        },

[dijital20]

For cryptography, there's a compiled component written in Rust which is packaged along with the Python code. This would be different per-platform, which accounts for the different hashes per platform. This page on PyPI shows the different platform variations of the files:

https://pypi.org/project/cryptography/#files

The lock you posted @matteius looks like it got all of them. What @xyxz-web is seeing are probably the ones appropriate for the platform and interpreter they're using.

Like I said, I have a colleague who is trying to author a new package and wants to support both Windows and Linux, so he's developing out of both Windows and WSL2 Ubuntu. At least one of the dependencies (I suspect lxml) is different between the platforms, so if he locks on one platform, the lock file doesn't work on the other and vice versa.

We have changed to using lock on Windows, and then lock --keep-outdated on the Linux, but there's got to be an easier, slightly more obvious way.

I hope this is helpful. :) Thank you for the discussion on this.

[matteius]

@dijital20 You are right that locked that on windows, but I just tried on Ubuntu and got the same result:

        "cryptography": {
            "hashes": [
                "sha256:093cb351031656d3ee2f4fa1be579a8c69c754cf874206be1d4cf3b542042804",
                "sha256:0cc20f655157d4cfc7bada909dc5cc228211b075ba8407c46467f63597c78178",
                "sha256:1b9362d34363f2c71b7853f6251219298124aa4cc2075ae2932e64c91a3e2717",
                "sha256:1f3bfbd611db5cb58ca82f3deb35e83af34bb8cf06043fa61500157d50a70982",
                "sha256:2bd1096476aaac820426239ab534b636c77d71af66c547b9ddcd76eb9c79e004",
                "sha256:31fe38d14d2e5f787e0aecef831457da6cec68e0bb09a35835b0b44ae8b988fe",
                "sha256:3b8398b3d0efc420e777c40c16764d6870bcef2eb383df9c6dbb9ffe12c64452",
                "sha256:3c81599befb4d4f3d7648ed3217e00d21a9341a9a688ecdd615ff72ffbed7336",
                "sha256:419c57d7b63f5ec38b1199a9521d77d7d1754eb97827bbb773162073ccd8c8d4",
                "sha256:46f4c544f6557a2fefa7ac8ac7d1b17bf9b647bd20b16decc8fbcab7117fbc15",
                "sha256:471e0d70201c069f74c837983189949aa0d24bb2d751b57e26e3761f2f782b8d",
                "sha256:59b281eab51e1b6b6afa525af2bd93c16d49358404f814fe2c2410058623928c",
                "sha256:731c8abd27693323b348518ed0e0705713a36d79fdbd969ad968fbef0979a7e0",
                "sha256:95e590dd70642eb2079d280420a888190aa040ad20f19ec8c6e097e38aa29e06",
                "sha256:a68254dd88021f24a68b613d8c51d5c5e74d735878b9e32cc0adf19d1f10aaf9",
                "sha256:a7d5137e556cc0ea418dca6186deabe9129cee318618eb1ffecbd35bee55ddc1",
                "sha256:aeaba7b5e756ea52c8861c133c596afe93dd716cbcacae23b80bc238202dc023",
                "sha256:dc26bb134452081859aa21d4990474ddb7e863aa39e60d1592800a8865a702de",
                "sha256:e53258e69874a306fcecb88b7534d61820db8a98655662a3dd2ec7f1afd9132f",
                "sha256:ef15c2df7656763b4ff20a9bc4381d8352e6640cfeb95c2972c38ef508e75181",
                "sha256:f224ad253cc9cea7568f49077007d2263efa57396a2f2f78114066fd54b5c68e",
                "sha256:f8ec91983e638a9bcd75b39f1396e5c0dc2330cbd9ce4accefe68717e6779e0a"
            ],
            "index": "pypi",
            "version": "==37.0.2"
        },

I am going to need y'all to provide the output from pipenv --support to further debug,

I also get the long list of hashes when I lock from within WSL - seems to include both Windows and Linux hashes.
However, when I lock from within Windows 10 or a Linux VM, I get a much shorter list.
I'm usinge a private repo here, but it is automatically mirrored from pypi.

[matteius]

@xyxz-web hash collection of all packages is a specific feature of pypi. There is no API into private pypi servers for collecting the set of hashes, and I suspect that is where the problem comes in. For reference, here is the code that collects the hashes: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L757-L788 Specifically look for where it collects hashes from pypi: https://github.com/pypa/pipenv/blob/main/pipenv/utils/resolver.py#L726-L755

I would recommend if possible to use pypi as a primary index in the Pipfile and support your private packages as a secondary index calling out specifically the private index to pull those specific packages from.