gh-133312: configure: add --enable-static-libpython-for-interpreter

[geofft]

This option changes the behavior of --enable-shared to continue to build the libpython3.x.so shared library, but not use it for linking the python3 interpreter executable. Instead, the executable is linked directly against the libpython .o files as it would be with --disable-shared.

There are two benefits of this change. First, libpython uses thread-local storage, which is noticeably slower when used in a loaded module instead of in the main program, because the main program can take advantage of constant offsets from the thread state pointer but loaded modules have to dynamically call a function __tls_get_addr() to potentially allocate their thread-local storage area. (There is another thread-local storage model for dynamic libraries which mitigates most of this performance hit, but it comes at the cost of preventing dlopen("libpython3.x.so"), which is a use case we want to preserve.)

Second, this improves the user experience around relocatable Python a little bit, in that we don't need to use an $ORIGIN-relative path to locate libpython3.x.so, which has some mild benefits around musl (which does not support $ORIGIN-relative DT_NEEDED, only $ORIGIN-relative DT_RPATH/DT_RUNPATH), users who want to make the interpreter setuid or setcap (which prevents processing $ORIGIN), etc.

• Issue: Enable building both an interpreter that statically links libpython and a shared library too #133312

---

📚 Documentation preview 📚: https://cpython-previews--133313.org.readthedocs.build/

[colesbury]

@geofft - I think you need to run autoconf and commit the modified configure file as well.

[geofft]

Thanks @AA-Turner and @colesbury!

Also cc @vstinner, you've made some other similar changes to configure.ac so this might be of interest to you.

[zanieb]

Thanks @geofft!

Also of note, the only way to get this behavior currently is to perform two builds and hack them together, which is what Debian does (or used to do?). Could you speak to that a bit? (I believe you have more context than me)

[geofft]

Yeah, I did a longer writeup in the issue, #133312, with some links to what Debian is currently doing.

[vstinner]

This option changes the behavior of --enable-shared to continue to build the libpython3.x.so shared library, but not use it for linking the python3 interpreter executable. Instead, the executable is linked directly against the libpython .o files as it would be with --disable-shared.

In this case, what's the use case for libpython? Is it to embed Python in an application? What happens if you load libpython in the python program?

[geofft]

In this case, what's the use case for libpython? Is it to embed Python in an application?

Yes, it's for embedding use cases (gdb with Python extensions, bundlers/compilers like Nuitka that produce a binary that links libpython, the test binary produced by cargo test in a pyo3 project, etc.). Admittedly many of these would probably have a better time using libpython.a for many of the same motivations as having python3 itself use it (performance, setuid/setcap, not having to think about rpaths or shipping libpython.so), but for backwards compatibility it's useful for distributors of Python that previously shipped a libpython.so to continue shipping it. (At least, that's why we want this in python-build-standalone instead of just dropping --enable-shared, and I assume that's also why Debian 23 years ago did the complicated thing. Also, I've struggled to use Debian's libpython.a but I do not yet understand in sufficient detail why it's hard.)

What happens if you load libpython in the python program?

How/why would you load it? ctypes.CDLL("libpython3.x.so.1") is going to get you weird behavior, but if you want to get at the C API from inside Python, ctypes.CDLL(None) is a better approach anyway: it works both with interpreters that link libpython.so and with those that don't. (And it's cross-platform and doesn't assume the filename, etc.)

Extension modules are supposed to not link libpython.so, because if they do, they will fail to load in a --disable-shared interpreter where there is no libpython.so for them to load. (If they do so anyway, and you try to load them in an environment where there is a libpython.so but the current interpreter doesn't use it, my guess is that the behavior is platform-dependent/undefined—probably with a dynamic linker without two-level namespaces (e.g., glibc) symbols will resolve to the main program and things will work, and with a dynamic linker with two-level namespaces (e.g. Mac dyld) symbols will resolve to the newly loaded libpython.so and things will break. I don't know if there's anything straightforward we can do technically to check for this, and I would argue it doesn't make sense to do so. Note also that this problem is not new; it would happen with current Debian Python, which ships effectively the same configuration as this option proposes, as well as all sorts of cases where you are running Python from some custom source but there happens to be a libpython.so somewhere on your library path.)

In other words, any Python code that loads libpython, either via ctypes etc. or via an extension module with a library dependency on libpython, will fail to work on a --disable-shared interpreter, and that is already supported and fairly common.

[ncoghlan]

Since option name bikeshedding was requested: rather than --enable-shared --enable-static-libpython-for-interpreter, perhaps the spelling could be --disable-shared --emit-shared?

That would reflect that the main interpreter binary isn't using the shared library, but the build process is being asked to emit the shared library anyway.

Setting --enable-shared would then implicitly set --emit-shared

[geofft]

Since option name bikeshedding was requested: rather than --enable-shared --enable-static-libpython-for-interpreter, perhaps the spelling could be --disable-shared --emit-shared?

So, my hesitation here is that --enable-shared is documented as "Enable building a shared Python library: libpython (default is no)." It is technically an undocumented side effect that it changes bin/python to depend on libpython—though I suspect that, in practice, this is a more common reason to use the option.

Therefore, doing something like --disable-shared --emit-shared would be redefining --enable-shared/--disable-shared to be about what bin/python does, and moving the thing it is documented as doing to a new option.

If we're all relatively confident that this is okay, then I can change the patch to do that, but that is why my current approach is to add a single new option.

Under this approach, --enable-shared would hard imply --emit-shared because you can't link a shared library unless you have one, and --disable-shared would default to --no-emit-shared or whatever we call it. So I guess it would be backwards compatible, and I'm feeling a little less worried about it than I was originally.