pyo3 build or run failed (ld -lpython) when using a python manager like `uv` or any `venv`

[Hennzau]

Hello,

I’ve faced quite a few challenges when building and running projects with PyO3 on systems that are fully managed by Python virtual environments. Most of the time, the issue boils down to errors like ld -lpython3.X not found. I wanted to share a workaround that worked for me.

On UNIX systems, you can see the specific directories ld searches by running the following command:

ld --verbose | grep SEARCH_DIR | tr -s ' ;' \\012

This will output something like:

SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu")
SEARCH_DIR("=/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu")
SEARCH_DIR("=/usr/lib/x86_64-linux-gnu64")
SEARCH_DIR("=/usr/local/lib64")
SEARCH_DIR("=/lib64")
SEARCH_DIR("=/usr/lib64")
SEARCH_DIR("=/usr/local/lib")
SEARCH_DIR("=/lib")
SEARCH_DIR("=/usr/lib")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib64")
SEARCH_DIR("=/usr/x86_64-linux-gnu/lib")

However, when using tools like uv, anaconda, etc., the Python installation is often standalone, and ld doesn’t know where to find the necessary libraries.

For example, with uv, the Python library is typically located in ~/.local/share/uv/python/cpython-{version}/lib. To fix this, you can configure ld to include this directory in its search path. For Python 3.13.1 on x86_64 Linux, you can do the following:

echo "/home/USER/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib" | sudo tee /etc/ld.so.conf.d/uv_python.conf

Then, reload the linker configuration using:

sudo ldconfig

After that, ld should be able to find the Python libraries correctly.

You can adapt this approach for any Python environment manager. I haven’t found clear documentation on this issue, so I hope this solution helps others who run into the same problem.

[ngoldbaum]

I wonder if we can fix this inside of pyo3-build-config somehow. I'd also like to add a CI job using setup-uv.

[mcmah309]

I ran into issue when using uv I wrote this this script to generate a build.rs that allows pyo3 projects to be built and run. Not sure how this could be incorporated into a permanent solution, but hopefully it can shed more light on the issue.

#!/usr/bin/env bash

. .venv/bin/activate

LIBPYTHON_PATH="$(ldd "$(readlink "$(which python)")" | grep 'libpython' | sed 's|^\(.*\)/libpython.*|\1|' | xargs)"
PYTHON_HOME="$(dirname "$(dirname "$(readlink "$(which python)")")")"

cat <<EOF > build.rs
fn main() {
    println!("cargo:rustc-env=LD_LIBRARY_PATH={}", "$LIBPYTHON_PATH");
    println!("cargo:rustc-env=PYTHONHOME={}", "$PYTHON_HOME");
}
EOF

echo "build.rs has been generated successfully."

out:

fn main() {
    println!("cargo:rustc-env=LD_LIBRARY_PATH={}", "/root/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu/bin/../lib");
    println!("cargo:rustc-env=PYTHONHOME={}", "/root/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu");
}

Without LIBPYTHON_PATH I get

error while loading shared libraries: libpython3.12.so.1.0: cannot open shared object file: No such file or directory

And without PYTHON_HOME I get

Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

[astrojuanlu]

Experiencing the same issue.

On macOS, it works when doing DYLD_FALLBACK_LIBRARY_PATH=/Users/juan_cano/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib CARGO_BUILD_RUSTFLAGS="-L /Users/j uan_cano/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/lib" cargo test (I need to specify both env vars)

On Linux, it works when doing LD_LIBRARY_PATH=/root/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/bin/../lib cargo test

[zanieb]

@astrojuanlu Are you using the latest version of uv? (and have you reinstalled the managed version with uv python install 3.13.0 --reinstall?) We added a patch to the macOS dynamic library in astral-sh/uv#10629

[astrojuanlu]

Indeed @zanieb , doing so made cargo test work without extra flags, thanks!

[geofft]

@Hennzau - I'm not able to reproduce this; the Python distribution installed by uv seems to work fine with pyo3. Can you provide instructions on how to reproduce this? (Or maybe something got fixed in either uv or pyo3 recently?)

$ cargo init --lib w
$ cd w
$ echo 'pyo3 = { version = "0.23" }' >> Cargo.toml 
$ cargo build
[...]
  --- stderr
  error: no Python 3.x interpreter found
$ PYO3_PYTHON=~/.local/share/uv/python/cpython-3.12.8-linux-aarch64-gnu/bin/python3.12 cargo build
[...]
    Finished dev [unoptimized + debuginfo] target(s) in 10.75s

See also my comment in astral-sh/uv#11006 (comment) .

[Hennzau]

@geofft You have probably some global python3.12 installed somewhere with the according libpython3.12, and it's why it's not failing to find it.

[Hennzau]

@geofft You can use this command to see where it's finding libpython3.12

ldconfig -p | grep libpython3.12

[geofft]

This machine does not have a global Python 3.12 anywhere. (It's the NixOS live CD, which doesn't have a global anything.)

Weirdly this is what I get:

[nixos@nixos:~]$ ldconfig -p | grep libpython3.12
ldconfig: Can't open cache file /nix/store/8mh0vrd6ll02j61612wz28vvsww1ps1z-glibc-2.39-52/etc/ld.so.cache
: No such file or directory

(There is also no /etc/ld.so.cache file.)

I also just tested the commands I provided on an Ubuntu 20.04 x86-64 machine and was unable to reproduce the issue either.

$ ldconfig -p | grep libpython
	libpython3.9.so.1.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.9.so.1.0
	libpython3.9.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.9.so
	libpython3.8.so.1.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0
	libpython3.8.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.8.so
	libpython3.7m.so.1.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0
	libpython3.7m.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython3.7m.so
	libpython2.7.so.1.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython2.7.so.1.0
	libpython2.7.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libpython2.7.so
$ curl -LSsf https://astral.sh/uv/install.sh | sh
$ . .local/bin/env
$ uv python install 3.12
$ cargo init --lib w
     Created library package
$ cd w
$ echo 'pyo3 = { version = "0.23" }' >> Cargo.toml 
$ PYO3_PYTHON=~/.local/share/uv/python/cpython-3.12.8-linux-x86_64-gnu/bin/python3.12 cargo build
[...]
    Finished dev [unoptimized + debuginfo] target(s) in 16.66s

Can you please provide some more details on what the exact error is that you are getting on your machine and how I can reproduce the problem?

From the linked uv issue - can I guess that your issue has to do with either a binary that links Python (as opposed to an extension module loaded by Python), or cargo test (which produces a binary that links Python)? Building an extension module that gets imported into a Python interpreter should work fine but I'd like to know if there's a case where it doesn't.

[mcmah309]

@geofft if I recall NixOs patches binaries since it does not use the standard file system structure. See https://github.com/nix-community/nix-ld .

If I had to guess, the issue here is same as astral-sh/uv#11006 and involves running a binary that tries to link to python at runtime

[Hennzau]

@geofft Hi again, something got fixed, probably, even if my libpython is not in the default search path of ld, even if I want to use a venv, PYO3 succeeds in finding libpython :

  -- PYO3_PRINT_CONFIG=1 is set, printing configuration and halting compile --
  implementation=CPython
  version=3.13
  shared=true
  abi3=false
  lib_name=python3.13
  lib_dir=/home/enzo/.local/share/uv/python/cpython-3.13.1-linux-x86_64-gnu/lib
  executable=/home/enzo/Downloads/w/.venv/bin/python
  pointer_width=64
  build_flags=
  python_framework_prefix=
  suppress_build_script_link_lines=false