Compatibility problems with `importlib.metadata` and `pypa/build`

[abravalheri]

Hello, recently I have been fighting some CI issues when trying to use crossenv for building a package using pypa/build¹.

I am having problems however to make build understand packages that have been exposed via cross-expose.
Specifically, build fails with a message ERROR Missing dependencies: <package that was supposed to be exposed by cross-expose>.

Please find a reproducer below:

> docker run --rm -it messense/manylinux2014-cross:aarch64 /bin/bash

#### Debug Info:
echo $TARGET_CC # => aarch64-unknown-linux-gnu-gcc
echo $TARGET_CXX # => aarch64-unknown-linux-gnu-g++
echo $TARGET_SYSROOT # => /usr/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/

#### Dummy project
mkdir /tmp/myproj
cd /tmp/myproj
touch dummy.py
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools", "cffi"]
EOF

#### Setup cross enviroment
python3.9 -m pip install crossenv
python3.9 -m crossenv "/opt/python/cp39-cp39/bin/python3" --cc $TARGET_CC --cxx $TARGET_CXX --sysroot $TARGET_SYSROOT --env LIBRARY_PATH= --manylinux manylinux1 .venv
source .venv/bin/activate
which build-pip # => /tmp/myproj/.venv/bin/build-pip
which pip # => /tmp/myproj/.venv/cross/bin/pip

#### We need build dependencies in cross-pip, but let's have them in build-pip (because of eventual environment leakages)
cross-pip install -U 'pip==23.2.1' 'setuptools==68.0.0' 'wheel==0.41.1' 'build==0.10.0'
build-pip install -U 'pip==23.2.1' 'setuptools==68.0.0' 'wheel==0.41.1' 'build==0.10.0'

build-pip install 'cffi==1.15.1'  # Fails to install with cross-pip so we have to use cross-expose
cross-expose cffi
cross-python -m pip list
# Package         Version
# --------------- -------
# build           0.10.0
# cffi            1.15.1
# packaging       23.1
# pip             23.2.1
# pycparser       2.21
# pyproject_hooks 1.0.0
# setuptools      68.0.0
# tomli           2.0.1
# wheel           0.41.1


#### Building the package
cross-python -m build --no-isolation
# * Getting build dependencies for sdist...
# running egg_info
# creating dummy.egg-info
# writing dummy.egg-info/PKG-INFO
# writing dependency_links to dummy.egg-info/dependency_links.txt
# writing top-level names to dummy.egg-info/top_level.txt
# writing manifest file 'dummy.egg-info/SOURCES.txt'
# reading manifest file 'dummy.egg-info/SOURCES.txt'
# writing manifest file 'dummy.egg-info/SOURCES.txt'
# 
# ERROR Missing dependencies:
#         cffi

This error is probably raised by https://github.com/pypa/build/blob/0.10.0/src/build/__main__.py#L131,
and I suspect that it is caused because cross-expose/crossenv is not compatible with importlib.metadata:

cross-python -c 'from importlib.metadata import distribution; distribution("cffi")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.9/importlib/metadata.py", line 542, in distribution
    return Distribution.from_name(distribution_name)
  File "/usr/lib/python3.9/importlib/metadata.py", line 196, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: cffi

Footnotes

1. I use pypa/build because since PEP 517 direct running python setup.py is considered deprecated.
   Instead users are advised to use a build frontend that follows PEP 517. ↩

[benfogle]

Your suspicions were correct. A patch is in master. Can you confirm if that fixes the error you're seeing?

[abravalheri]

Thank you very much @benfogle. I tested the following script and it seems to be fine:

> docker run --rm -it messense/manylinux2014-cross:aarch64 /bin/bash

#### Debug Info:
echo $TARGET_CC       # => aarch64-unknown-linux-gnu-gcc
echo $TARGET_CXX      # => aarch64-unknown-linux-gnu-g++
echo $TARGET_SYSROOT  # => /usr/aarch64-unknown-linux-gnu/aarch64-unknown-linux-gnu/sysroot/

#### Dummy project
mkdir /tmp/myproj
cd /tmp/myproj
touch dummy.py
cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools", "cffi"]
EOF

#### Setup cross enviroment
python3.9 -m pip install https://github.com/benfogle/crossenv/archive/refs/heads/master.zip
python3.9 -m crossenv "/opt/python/cp39-cp39/bin/python3" --cc $TARGET_CC --cxx $TARGET_CXX --sysroot $TARGET_SYSROOT --env LIBRARY_PATH= --manylinux manylinux1 .venv
source .venv/bin/activate
which build-pip # => /tmp/myproj/.venv/bin/build-pip
which pip # => /tmp/myproj/.venv/cross/bin/pip

#### We need build dependencies in cross-pip, but let's have them in build-pip (because of eventual environment leakages)
cross-pip install -U 'pip==23.2.1' 'setuptools==68.0.0' 'wheel==0.41.1' 'build==0.10.0'
build-pip install -U 'pip==23.2.1' 'setuptools==68.0.0' 'wheel==0.41.1' 'build==0.10.0'

build-pip install 'cffi==1.15.1'  # Fails to install with cross-pip so we have to use cross-expose
cross-expose cffi
cross-python -m pip list
# Package         Version
# --------------- -------
# build           0.10.0
# cffi            1.15.1
# packaging       23.1
# pip             23.2.1
# pycparser       2.21
# pyproject_hooks 1.0.0
# setuptools      68.0.0
# tomli           2.0.1
# wheel           0.41.1

#### Building the package
cross-python -m build --no-isolation
# ...
# Successfully built dummy-0.0.0.tar.gz and dummy-0.0.0-py3-none-any.whl

------

## Inside the same container test something closer to the CI job in setuptools-rust
curl -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
source ~/.cargo/env
rustup target add aarch64-unknown-linux-gnu
git clone https://github.com/PyO3/setuptools-rust.git /tmp/setuptools-rust
cross-pip install /tmp/setuptools-rust

export DIST_EXTRA_CONFIG=/tmp/build-opts.cfg
echo -e "[bdist_wheel]\npy_limited_api=cp37" > $DIST_EXTRA_CONFIG
cross-python -m build --no-isolation //tmp/setuptools-rust/examples/rust_with_cffi
# ...
# Successfully built rust-with-cffi-0.1.0.tar.gz and rust_with_cffi-0.1.0-cp37-abi3-linux_aarch64.whl