spin build should fix the rpath for shared libaries

[oscarbenjamin]

I'm migrating python-flint to meson-python and testing out spin as a frontend (flintlib/python-flint#129).

The setup that I use with python-flint is to build libraries with --prefix=$(pwd)/.local so that I have isolated builds of the C dependencies inside the python-flint development checkout.

When building wheels with meson-python I can set PKG_CONFIG_PATH to find the libraries and then meson-python fixes the rpath:
https://github.com/mesonbuild/meson-python/blob/c9d1c2dafb54692ad56dcda1bc67fd52b83b29c0/mesonpy/__init__.py#L431-L440

This means that I can do e.g.

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig spin install
 $ pip install --no-build-isolation --editable .
 ...
Successfully installed python-flint-0.6.0
$ python -m flint.test -qt
Running tests...
flint.test: all 49 tests passed!
----------------------------------------
OK: Your installation of python-flint seems to be working just fine!
----------------------------------------

However spin build does not fix the rpath and so spin test does not work unless I set LD_LIBRARY_PATH:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig spin build --clean
...
[90/90] Linking target src/flint/functions/showgood.cpython-312-x86_64-linux-gnu.so
 $ meson install --only-changed -C build --destdir ../build-install
$ spin test
Invoking `build` prior to running tests:
 $ meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /usr/bin/ninja -C /home/oscar/current/active/python-flint/build
ninja: Entering directory `/home/oscar/current/active/python-flint/build'
ninja: no work to do.
 $ meson install --only-changed -C build --destdir ../build-install
 $ export PYTHONPATH="/home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages"
 $ /home/oscar/.pyenv/versions/3.12.0/envs/python-flint-3.12/bin/python3.12 -P -c 'import flint'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages/flint/__init__.py", line 1, in <module>
    from .pyflint import *
  File "pyflint.pyx", line 1, in init flint.pyflint
  File "flint_base.pyx", line 1, in init flint.flint_base.flint_base
ImportError: libflint.so.19: cannot open shared object file: No such file or directory
As a sanity check, we tried to import flint.
Stopping. Please investigate the build error.
$ LD_LIBRARY_PATH=$(pwd)/.local/lib spin test
Invoking `build` prior to running tests:
 $ meson compile -C build
INFO: autodetecting backend as ninja
INFO: calculating backend command to run: /usr/bin/ninja -C /home/oscar/current/active/python-flint/build
ninja: Entering directory `/home/oscar/current/active/python-flint/build'
ninja: no work to do.
 $ meson install --only-changed -C build --destdir ../build-install
 $ export PYTHONPATH="/home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages"
 $ /home/oscar/.pyenv/versions/3.12.0/envs/python-flint-3.12/bin/python3.12 -P -c 'import flint'
 $ export PYTHONPATH="/home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages"
 $ cd /home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages
 $ /home/oscar/.pyenv/versions/3.12.0/envs/python-flint-3.12/bin/python3.12 -m pytest --rootdir=/home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages flint
======================================== test session starts ========================================
platform linux -- Python 3.12.0, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/oscar/current/active/python-flint/build-install/usr/lib/python3.12/site-packages
plugins: doctestplus-1.2.1, cov-4.1.0, hypothesis-6.88.4, xdist-3.4.0
collected 49 items                                                                                  

flint/test/test_stuff.py .................................................                    [100%]

======================================== 49 passed in 3.86s =========================================

I think that spin build should set the rpath somehow but I'm not sure where is the place that this should be fixed. I assume that it would be out of place for spin to manipulate the rpath entries directly like meson-python does.

Should meson set the rpath?

Should spin ask meson or meson-python to do it somehow?

[rgommers]

Since editable installs already work for you, gh-155 to make editable installs work with spin should be helpful here.

For the default spin build behavior, I'm actually not 100% sure what is happening.

When building wheels with meson-python I can set PKG_CONFIG_PATH to find the libraries and then meson-python fixes the rpath:

This is probably not the relevant code, because it's about internal shared libraries, while the ones you have are external (not built as part of the meson setup / meson compile invocation as a subproject) ones, right?

Normally Meson will take care of rpath rewriting during install. It's possible that that doesn't work for in-tree shared libraries because they're not considered external. Does it work without LD_LIBRARY_PATH if you place them elsewhere, like $(pwd)/../.local/lib?

The entries should have absolute paths for shared library dependencies that are not on the default loader search path. For example:

# spin build folder
$ ldd build/scipy/linalg/_fblas.cpython-311-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffeee2fd000)
        libopenblas.so.0 => /home/rgommers/mambaforge/envs/scipy-dev/lib/libopenblas.so.0 (0x000078fd15e00000)
        libc.so.6 => /usr/lib/libc.so.6 (0x000078fd15bfb000)
        libm.so.6 => /usr/lib/libm.so.6 (0x000078fd15b0f000)

# spin install folder
$ ldd build-install/lib/python3.11/site-packages/scipy/linalg/_fblas.cpython-311-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffcb6baa000)
        libopenblas.so.0 => /home/rgommers/mambaforge/envs/scipy-dev/lib/libopenblas.so.0 (0x00007311fc600000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007311fc41e000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007311fc332000)

RPATH entry:

$ readelf -d build/scipy/linalg/_fblas.cpython-311-x86_64-linux-gnu.so | rg RPATH
 0x000000000000000f (RPATH)              Library rpath: [/home/rgommers/mambaforge/envs/scipy-dev/lib]

[oscarbenjamin]

the ones you have are external (not built as part of the meson setup / meson compile invocation as a subproject) ones, right?

Yes, that's correct.

This is what I get from ldd:

# build dir
$ ldd build/src/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffd673d8000)
	libflint.so.19 => /home/oscar/current/active/python-flint/build/src/flint/types/../../../../.local/lib/libflint.so.19 (0x0000737a77000000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000737a76c00000)
	libmpfr.so.6 => /home/oscar/current/active/python-flint/.local/lib/libmpfr.so.6 (0x0000737a76800000)
	libgmp.so.10 => /home/oscar/current/active/python-flint/.local/lib/libgmp.so.10 (0x0000737a76f5b000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x0000737a76e74000)
	/lib64/ld-linux-x86-64.so.2 (0x0000737a77c8f000)

# install dir
$ ldd build-install/usr/lib/python3.12/site-packages/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007ffef73fd000)
	libflint.so.19 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000709664000000)
	/lib64/ld-linux-x86-64.so.2 (0x000070966440b000)

This is what readelf shows:

$ readelf -d build/src/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so

Dynamic section at offset 0x41cc8 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libflint.so.19]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../../.local/lib]
 0x000000000000000c (INIT)               0x6000
 0x000000000000000d (FINI)               0x37de4
 0x0000000000000019 (INIT_ARRAY)         0x42cb8
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x42cc0
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x2f0
 0x0000000000000005 (STRTAB)             0x1878
 0x0000000000000006 (SYMTAB)             0x318
 0x000000000000000a (STRSZ)              4012 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x43000
 0x0000000000000002 (PLTRELSZ)           4392 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4ad8
 0x0000000000000007 (RELA)               0x2a20
 0x0000000000000008 (RELASZ)             8376 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x29f0
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x2824
 0x000000006ffffff9 (RELACOUNT)          306
 0x0000000000000000 (NULL)               0x0
$ readelf -d build-install/usr/lib/python3.12/site-packages/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so 

Dynamic section at offset 0x41cc8 contains 25 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libflint.so.19]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000000c (INIT)               0x6000
 0x000000000000000d (FINI)               0x37de4
 0x0000000000000019 (INIT_ARRAY)         0x42cb8
 0x000000000000001b (INIT_ARRAYSZ)       8 (bytes)
 0x000000000000001a (FINI_ARRAY)         0x42cc0
 0x000000000000001c (FINI_ARRAYSZ)       8 (bytes)
 0x000000006ffffef5 (GNU_HASH)           0x2f0
 0x0000000000000005 (STRTAB)             0x1878
 0x0000000000000006 (SYMTAB)             0x318
 0x000000000000000a (STRSZ)              4012 (bytes)
 0x000000000000000b (SYMENT)             24 (bytes)
 0x0000000000000003 (PLTGOT)             0x43000
 0x0000000000000002 (PLTRELSZ)           4392 (bytes)
 0x0000000000000014 (PLTREL)             RELA
 0x0000000000000017 (JMPREL)             0x4ad8
 0x0000000000000007 (RELA)               0x2a20
 0x0000000000000008 (RELASZ)             8376 (bytes)
 0x0000000000000009 (RELAENT)            24 (bytes)
 0x000000006ffffffe (VERNEED)            0x29f0
 0x000000006fffffff (VERNEEDNUM)         1
 0x000000006ffffff0 (VERSYM)             0x2824
 0x000000006ffffff9 (RELACOUNT)          306
 0x0000000000000000 (NULL)               0x0

I'm not sure why I have an extra usr like build-install/usr/.... When building with spin build I get:

  User defined options
    prefix: /usr

I don't think I've set that anywhere though...

[rgommers]

Library runpath: [$ORIGIN/../../../../.local/lib]

Yep, this being a relative rather than an absolute path should be the problem.

I don't think I've set that anywhere though...

That's the output of meson, so "user defined" here is something set by spin.

[oscarbenjamin]

Does it work without LD_LIBRARY_PATH if you place them elsewhere, like $(pwd)/../.local/lib?

No, it still doesn't work in that case. I just get an extra ..:

 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../../../.local/lib]

[oscarbenjamin]

No, it still doesn't work in that case. I just get an extra ..:

To be clear I used an absolute path to a directory outside of the working directory

$ PKG_CONFIG_PATH=/home/oscar/current/active/.local/lib/pkgconfig/ spin build --clean

but it somehow got turned into a relative path.

[oscarbenjamin]

So cutting out spin then the problem is just:

$ PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig meson setup build
$ meson compile -C build
$ readelf -d build/src/flint/types/fmpz.cpython-312-x86_64-linux-gnu.so | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../../.local/lib]

We wanted an absolute path to be in the binary but we've been left with a relative one.

Should that be considered a bug in meson?

[oscarbenjamin]

It all works fine on macos so I'm only seeing this problem on Linux.

[stefanv]

@oscarbenjamin If I understand this issue correctly, this is a Meson problem. If you agree, we can close this, otherwise I can investigate further.

[oscarbenjamin]

Yeah, this is just a meson issue I think.

[oscarbenjamin]

The solution to this (mesonbuild/meson#13046 (comment)) is to set LDFLAGS when calling spin build i.e.:

PKG_CONFIG_PATH=$(pwd)/.local/lib/pkgconfig LDFLAGS=-Wl,-rpath=$(pwd)/.local/lib spin build

Possibly it would be useful for spin to add a more discoverable CLI option for that like

spin build --local-lib-dir=.local/lib

In flintlib/python-flint#135 I added a custom option spin build --local-flint=.local/lib/pkgconfig which translates to

spin build -- --pkg-config-path=$(pwd)/.local/lib/pkgconfig -Dlocal_flint=true

The local_flint project option sets the linker flags to install the rpaths.

[oscarbenjamin]

From the discussion in mesonbuild/meson#13046 I think that this is sort of a spin-specific situation because of the fact that we are "installing" things into a local directory.