Using pysqlite3-binary fails on `install --deploy` when locked on M1 and installed on Linux

[nk9]

Issue description

I have a Pipfile which has two requirements in it, pysqlite3-binary and pytz. Either requirement installs successfully on its own, but pipenv fails when asked to install them together. In testing, I've discovered that it appears that ANY package paired with pysqlite3-binary will cause the install to fail.

Expected result

I expected the install to work. This same setup worked last year, but when I tried to do it again, it failed this time.

Actual result

Build log

 > [python-deps 5/5] RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose:
#10 4.731 Using python: 3.10
#10 4.731 Path to python: /var/lang/bin/python3
#10 4.731 Creating a virtualenv for this project...
#10 4.731 Pipfile: /var/task/Pipfile
#10 4.917 Using /var/lang/bin/python3 (3.10.11) to create virtualenv...
#10 8.962 created virtual environment CPython3.10.11.final.0-64 in 2898ms
#10 8.962   creator CPython3Posix(dest=/var/task/.venv, clear=False, no_vcs_ignore=False, global=False)
#10 8.962   seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/root/.local/share/virtualenv)
#10 8.962     added seed packages: pip==23.1.2, setuptools==67.7.2, wheel==0.40.0
#10 8.962   activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
#10 8.962
#10 8.994 ✔ Successfully created virtual environment!
#10 9.396 Virtualenv location: /var/task/.venv
#10 9.734 Installing dependencies from Pipfile.lock (36200f)...
#10 9.799 Writing supplied requirement line to temporary file: "pysqlite3-binary ; platform_machine == 'x86_64'"
#10 9.823 Writing supplied requirement line to temporary file: 'pytz==2023.3 --hash=sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588 --hash=sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb'
#10 9.827 Install Phase: Standard Requirements
#10 9.827 Preparing Installation of 'pysqlite3-binary'
#10 9.827 Preparing Installation of 'pytz'
#10 9.828 $ /var/task/.venv/bin/python /var/lang/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-deps --exists-action=i -r /tmp/pipenv-_poa8qn4-requirements/pipenv-manv2i0z-hashed-reqs.txt
#10 9.828 Using source directory: '/var/task/.venv/src'
#10 13.90 Collecting pysqlite3-binary (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-manv2i0z-hashed-reqs.txt (line 1))
#10 13.90
#10 14.17 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#10 14.17     pysqlite3-binary from https://files.pythonhosted.org/packages/e5/af/750aa272a8282f6518a8532f31c51cc9e49815c626cf0df8cdf181367b52/pysqlite3_binary-0.5.0-cp310-cp310-manylinux_2_24_x86_64.whl (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-manv2i0z-hashed-reqs.txt (line 1))
#10 14.17 An error occurred while installing pysqlite3-binary ; platform_machine == 'x86_64'! Will try again.
#10 14.17 An error occurred while installing pytz==2023.3 --hash=sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588 --hash=sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb! Will try again.
#10 14.18 Installing initially failed dependencies...
#10 14.20 Writing supplied requirement line to temporary file: "pysqlite3-binary ; platform_machine == 'x86_64'"
#10 14.20 Writing supplied requirement line to temporary file: 'pytz==2023.3 --hash=sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588 --hash=sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb'
#10 14.20 Install Phase: Standard Requirements
#10 14.20 Preparing Installation of 'pysqlite3-binary'
#10 14.20 Preparing Installation of 'pytz'
#10 14.20 $ /var/task/.venv/bin/python /var/lang/lib/python3.10/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-use-pep517 --no-deps --exists-action=i -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt
#10 14.20 Using source directory: '/var/task/.venv/src'
#10 18.22 Collecting pysqlite3-binary (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt (line 1))
#10 18.22
#10 18.49 ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#10 18.49     pysqlite3-binary from https://files.pythonhosted.org/packages/e5/af/750aa272a8282f6518a8532f31c51cc9e49815c626cf0df8cdf181367b52/pysqlite3_binary-0.5.0-cp310-cp310-manylinux_2_24_x86_64.whl (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt (line 1))
#10 18.50 [pipenv.exceptions.InstallError]: ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
#10 18.50 [pipenv.exceptions.InstallError]:     pysqlite3-binary from https://files.pythonhosted.org/packages/e5/af/750aa272a8282f6518a8532f31c51cc9e49815c626cf0df8cdf181367b52/pysqlite3_binary-0.5.0-cp310-cp310-manylinux_2_24_x86_64.whl (from -r /tmp/pipenv-_poa8qn4-requirements/pipenv-v45mta1p-hashed-reqs.txt (line 1))
#10 18.50 ERROR: Couldn't install package: [Requirement(_name='pysqlite3-binary', vcs=None, req=NamedRequirement(name='pysqlite3-binary', version='', req=Requirement.parse('pysqlite3-binary; platform_machine == "x86_64"'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=pysqlite3-binary, path=None, uri=None, extras=(), markers=platform_machine == 'x86_64', vcs=None, specifier=None, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pysqlite3-binary)>), markers='platform_machine == "x86_64"', _specifiers='', index=None, editable=False, hashes=frozenset(), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=pysqlite3-binary, path=None, uri=None, extras=(), markers=platform_machine == 'x86_64', vcs=None, specifier=None, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pysqlite3-binary)>, _ireq=None), Requirement(_name='pytz', vcs=None, req=NamedRequirement(name='pytz', version='==2023.3', req=Requirement.parse('pytz==2023.3'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=pytz, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2023.3, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pytz==2023.3)>), markers=None, _specifiers='==2023.3', index='pypi', editable=False, hashes=frozenset({'sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb', 'sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588'}), extras=(), abstract_dep=None, _line_instance=<Line (editable=False, name=pytz, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===2023.3, pyproject=None, pyproject_requires=None, pyproject_backend=None, ireq=pytz==2023.3)>, _ireq=None)]
#10 18.50  Package installation failed...
------
executor failed running [/bin/sh -c PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose]: exit code: 1

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

Take this Dockerfile:

FROM public.ecr.aws/lambda/python:3.10 AS base

# Set up environment
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONFAULTHANDLER 1



FROM base AS python-deps

RUN pip install --upgrade pip
RUN pip install pipenv

COPY Pipfile* ./

RUN pipenv --support >> /tmp/pipenv-support.txt
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose

And this Pipfile:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
pytz = "*"
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }

[requires]
python_version = "3.10"

And then attempt to build the Dockerfile:

$ pipenv update # To create the lockfile
$ DOCKER_DEFAULT_PLATFORM=linux/amd64 docker build .

NB: the pysqlite3-binary package only has Linux wheels, hence the platform specifics. Also, I have tried both version = "*" and pinning the version to 0.5.0, but get the same failure each time.

I'm on an M1 Mac, although this same error is happening on GitHub Actions when it tries to build the container there.

---

$ pipenv --support

Pipenv version: '2023.6.2'

Pipenv location: '/var/lang/lib/python3.10/site-packages/pipenv'

Python location: '/var/lang/bin/python3.10'

OS Name: 'posix'

User pip version: '23.1.2'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.10.11',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.10.76-linuxkit',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Mon Nov 8 11:22:26 UTC 2021',
 'python_full_version': '3.10.11',
 'python_version': '3.10',
 'sys_platform': 'linux'}

System environment variables:

• _
• HOME
• SHLVL
• PYTHONDONTWRITEBYTECODE
• TZ
• LAMBDA_RUNTIME_DIR
• LANG
• PYTHONFAULTHANDLER
• PWD
• PATH
• LAMBDA_TASK_ROOT
• LD_LIBRARY_PATH
• LC_ALL
• PIP_DISABLE_PIP_VERSION_CHECK
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin
• LANG: C.UTF-8
• PWD: /var/task

---

Contents of Pipfile ('/var/task/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
# requests = "*"
# boto3 = "*"
# jinja2 = "*"
pytz = "*"
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }

# [dev-packages]
# isort = "*"
# black = "*"
# flake8 = "*"

[requires]
python_version = "3.10"

Contents of Pipfile.lock ('/var/task/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "2b1a32fdca8e7cac9c7b8bc7db8cd7219f6ff51cd0ed4002293e5ff66d36200f"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.10"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "pysqlite3-binary": {
            "platform_machine": "== 'x86_64'",
            "version": "*"
        },
        "pytz": {
            "hashes": [
                "sha256:1d8ce29db189191fb55338ee6d0387d82ab59f3d00eac103412d64e0ebd0c588",
                "sha256:a151b3abb88eda1d4e34a9814df37de2a80e301e68ba0fd856fb9b46bfbbbffb"
            ],
            "index": "pypi",
            "version": "==2023.3"
        }
    },
    "develop": {}
}

[nk9]

I think the problem here is that I'm developing on an M1 Mac but deploying on a Linux container. Since the Mac ignores the pysqlite3-binary package (wrong platform), the Pipfile.lock contains no hashes for that package. Then, I try to install using the lockfile in the Linux container, and the platform matches but the hashes are missing and so the install fails.

I have worked around this by locking within Docker like so:

-COPY Pipfile* ./
+COPY Pipfile ./

-RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose
+RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --verbose

However, this goes against the pipenv recommendtion to avoid modifying the lockfile during CI. So I would prefer another solution.

I also tried this approach of locking in the CI using --keep-outdated:

RUN PIPENV_VENV_IN_PROJECT=1 pipenv lock --keep-outdated
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --verbose

But I got the same failure as before for some reason.

Lastly, I tried using packages categories to restrict the locking to just the sqlite package, like so:

[packages]
requests = "*"
boto3 = "*"
jinja2 = "*"
pytz = "*"

[sqlite]
pysqlite3-binary = { version = "*", platform_machine = "== 'x86_64'" }

[dev-packages]
isort = "*"
black = "*"
flake8 = "*"

I then lock just that category in the Dockerfile:

COPY Pipfile* ./

RUN PIPENV_VENV_IN_PROJECT=1 pipenv lock --categories sqlite
RUN PIPENV_VENV_IN_PROJECT=1 pipenv install --deploy --categories="packages sqlite" --verbose

This works, and seems like the least bad option. But please let me know if I'm missing something.

[matteius]

This works, and seems like the least bad option. But please let me know if I'm missing something.

Yeah, this is probably your least bad option for now. The fundamental problem of building an sdist and getting the hashes for a different system than the one you are locking on is a well known problem.

[nk9]

Alright, thanks for confirming. Would be great to get a cleaner solution, but I'll use the multiple categories for now.