Spindle changes library search behavior, causes libmpi to not be found

[nchaimov]

In the Ubuntu 24.04 images used for CI, if MPICH is installed but ldconfig is not run to update the cache, MPI applications will run successfully without Spindle, but with Spindle at level medium or high, libmpi is not found:

./mpi-hello: error while loading shared libraries: libmpi.so.12: cannot open shared object file: No such file or directory

LD_DEBUG=all output without Spindle, ldconfig not run:

file=libmpi.so.12 [0];  needed by /shared/examples/./mpi-hello [0]
find library=libmpi.so.12 [0]; searching
 search path=/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib       (system search path)
  trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
  trying file=/usr/lib/glibc-hwcaps/x86-64-v2/libmpi.so.12
  trying file=/usr/lib/libmpi.so.12

file=libmpi.so.12 [0];  generating link map
  dynamic: 0x000077e828e7c8c0  base: 0x000077e8263d5000   size: 0x0000000002ea1d10
    entry: 0x000077e8263d5000  phdr: 0x000077e8263d5040  phnum:                 12

With Spindle, ldconfig run:

file=libmpi.so.12 [0];  needed by ./mpi-hello [0]
find library=libmpi.so.12 [0]; searching
 search path=/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib       (system search path)
  trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
  trying file=/__not_exglibc-hwcaps/x86-64-v2/libmpi.so.12
  trying file=/__not_exlibmpi.so.12
 search cache=/etc/ld.so.cache
  trying file=/usr/lib/libmpi.so.12

file=libmpi.so.12 [0];  generating link map
  dynamic: 0x00007a2974da58c0  base: 0x00007a29722fe000   size: 0x0000000002ea1d10
    entry: 0x00007a29722fe000  phdr: 0x00007a29722fe040  phnum:                 12

While libmpi.so.12 is found in the cache, this behavior is already different from the non-Spindle case, which finds libmpi.so.12 prior to searching the cache.

With Spindle, ldconfig not run, failing case:

file=libmpi.so.12 [0];  needed by ./mpi-hello [0]
find library=libmpi.so.12 [0]; searching
  search path=/usr/lib/glibc-hwcaps/x86-64-v3:/usr/lib/glibc-hwcaps/x86-64-v2:/usr/lib       (system search path)
  trying file=/usr/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
  trying file=/__not_exglibc-hwcaps/x86-64-v2/libmpi.so.12
  trying file=/__not_exlibmpi.so.12
 search cache=/etc/ld.so.cache
 search path=/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3:/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v2:/usr/lib/x8-gnu:/lib/hwcaps/x86-64-v3:/lib/glibc-hwcaps/x86-64-v2:/lib     (system search path)
  trying file=/lib/x86_64-linux-gnu/libmpi.so.12
  trying file=/usr/lib/x86_64-linux-gnu/glibc-hwcaps/x86-64-v3/libmpi.so.12
  trying file=/__not_exists/4-linux-gnu/glibc-hwcaps/x86-64-v2/libmpi.so.12
  trying file=/__not_exists/4-linux-gnu/libmpi.so.12
  trying file=/lib/glibc-hwcaps/x86-64-v3/libmpi.so.12
  trying file=/__noglibc-hwcaps/x86-64-v2/libmpi.so.12
  trying file=/__nolibmpi.so.12

./mpi-hello: error while loading shared libraries: libmpi.so.12: cannot open shared object file: No such file or directory

Here, /usr/lib doesn't get searched at all.