Embeded sub-interpreters

[b-pass]

Description

This PR adds a header file, subinterpreter.h which provides utility classes for embedded Python to create, use, and destroy sub-interpreters. The PR includes documentation rewrite of the "Embedded Sub-interpreter support" section. The PR also splits out the multiple interpreter tests from test_embed/test_interpreter.cpp into their own file in that directory, updates them to use the new API, and adds new tests to cover the API and some interesting aspects of subinterpreters.

API

Everything is in subinterpreter.h because this feature requires PYBIND11_SUBINTERPRETER_SUPPORT, which is 3.12+ (and some other caveats). Including this file in non-supported versions produces a #error. There are no notable API outside of this new file, except for a change to embedded modules which was split out into its own PR: #5665 . This PR currently includes the embed.h changes from that PR because this PR would not compile without them, and so this PR depends on that one being merged first (and when it is, I will fix this one with a rebase).

Class subinterpreter manages the lifetime of a sub-interpreter. Default constructing a subinterpreter does not create a subinterpreter it creates an empty wrapper, this is done to make the objects easier to move around and to prevent accidentally creating them (as creating them has to acquire and release the GIL).

To create a sub-interpreter, one does: py::subinterpreter si = subinterpreter::create(), similar to py::initialize_interpreter() for the main interpreter. I didn't provide a py::subinterpreter::destroy(...) since that doesn't seem to make a lot of sense (you instead destruct the subinterpreter object returned by create) but if we want one to mirror py::finalize_interpreter() then I could add one easily.

subinterpreter also includes some useful utilities: current() to get a non-owning subinterpreter for the currently active interpreter (even the main one), main() to get a non-owning subinterpreter for the main interpreter (even if it is not the current one), id() to get the ID number, and state_dict() to get the interpreter state dict.

Class subinterpreter_scoped_activate is the RAII wrapper, similar to gil_scoped_acquire which acquires the sub-interpreter's GIL and makes it the active interpreter. The similarity to what the gil RAII classes do is also why it is named that.

Class scoped_subinterpreter is also provided in order to mirror the main interpreter's scoped_interpreter.

Suggested changelog entry:

* Added API in ``pybind11/subinterpreter.h`` for embedding sub-intepreters (requires Python 3.12+).

---

📚 Documentation preview 📚: https://pybind11--5666.org.readthedocs.build/

[henryiii]

I think #5646 is about to go in, then I'll rebase this and add 3.13t and 3.14t in. (Edit: I think I meant to put this on the previous PR, didn't realize there were two)

[b-pass]

It looks to me like the 3.14 crashes are actually a CPython bug, python/cpython#134144.

[henryiii]

Do you know why mingw is failing?

[b-pass]

Do you know why mingw is failing?

Not sure. My guess is the failure is destroying the sub-interpreter on a different OS thread than the one that created it. You can do that in 3.13, but in 3.12 it seems even though the feature is "stable" there are some issue with PyThreadState objects on sub-interpreters in that version.

[rwgk]

@henryiii Maybe it's a good time to drop AppVeyor? I haven't seen anything resulting uniquely from the AppVeyor job for years.

[henryiii]

I don't think the failure is due to this PR, pasting it here and rerunning:

=================================== FAILURES ===================================
_ test_run_in_process_multiple_threads_parallel[test_cross_module_gil_nested_pybind11_released] _

test_fn = <function test_cross_module_gil_nested_pybind11_released at 0x2ebaf8a7500>

    @pytest.mark.skipif(sys.platform.startswith("emscripten"), reason="Requires threads")
    @pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK)
    @pytest.mark.skipif(
        "env.GRAALPY",
        reason="GraalPy transiently complains about unfinished threads at process exit",
    )
    def test_run_in_process_multiple_threads_parallel(test_fn):
        """Makes sure there is no GIL deadlock when running in a thread multiple times in parallel.
    
        It runs in a separate process to be able to stop and assert if it deadlocks.
        """
>       assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=True) == 0
E       assert -11 == 0
E        +  where -11 = _run_in_process(_run_in_threads, <function test_cross_module_gil_nested_pybind11_released at 0x2ebaf8a7500>, num_threads=8, parallel=True)

test_fn    = <function test_cross_module_gil_nested_pybind11_released at 0x2ebaf8a7500>

../../tests/test_gil_scoped.py:241: AssertionError
=============================== warnings summary ===============================
<frozen importlib._bootstrap>:491
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'exo_planet_c_api', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

[henryiii]

Yes, it's a flake, and unrelated. We'll need to see how often it shows up, and in which jobs (3.14t only, or both).

[Copilot]

Pull Request Overview

This PR introduces support for Python sub-interpreters in pybind11 by adding a dedicated API, updating documentation, and refactoring tests to exercise the new functionality.

• Added a new pybind11/subinterpreter.h header with RAII wrappers and utilities for sub-interpreters.
• Updated embedding documentation (docs/advanced/embedding.rst) to describe sub-interpreter API and best practices.
• Refactored existing interpreter tests: extracted sub-interpreter tests into test_subinterpreter.cpp and updated build configuration.

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/test_embed/test_interpreter.cpp	Removed embedded sub-interpreter tests and helper functions.
tests/test_embed/CMakeLists.txt	Added test_subinterpreter.cpp to the test executable.
tests/extra_python_package/test_files.py	Included include/pybind11/subinterpreter.h in test package.
include/pybind11/subinterpreter.h	New API for creating, using, and destroying Python sub-interpreters.
include/pybind11/gil.h	Clarified shutdown comment in gil_scoped_acquire.
docs/advanced/misc.rst	Added an anchor for concurrency section.
docs/advanced/embedding.rst	Rewrote embedding guide to cover sub-interpreter usage and safety.
CMakeLists.txt	Installed subinterpreter.h alongside other headers.

Comments suppressed due to low confidence (1)

docs/advanced/embedding.rst:484

• Use “its” instead of “it's” in documentation.

:class:`subinterpreter_scoped_activate` past the lifetime of it's :class:`subinterpreter`)

[henryiii]

@ericsnowcurrently You might be interested in this PR. ;)

[henryiii]

I'm dropping the flaky 3.14t free-threading logs into #5674. I've seen two of these now. They have not been in the subintepreters part (at least yet).

[ericsnowcurrently]

I started looking at the PR a few days ago and got distracted by other stuff. I do plan on getting back to it, but don't wait for me.

@henryiii (or any other pybind11 folks at PyCon), if you have time at the sprints I wouldn't mind walking through this with you.

[henryiii]

I left yesterday afternoon. @virtuald, are you still at the sprints?

The good news is the API is mostly separate from pybind11. What you do once you have a subinterpreter, of course, then you use pybind11 to interact with it, but the launching of subintepreters is isolated to this PR. (The embedding of Python is also from pybind11.)

I'm tempted to drop this into the 3.0 RC, since I'm still finishing preparations, and then we'll have ~1 week to change if anything pops up; this is new so last RC changes are fine, IMO.

[Skylion007]

Some comments

[Skylion007]

Suggested change

	subinterpreter &operator=(subinterpreter &&old) {
	subinterpreter &operator=(subinterpreter &&old) noexcept {

I'm surprised clang-tidy didn't catch this...

[Skylion007]

Hmm, is this not being included in the cmake for some reason?

[henryiii]

I don't think our embedded tests are. The header is included, but it doesn't get checked unless it's also compiled somewhere, I think.

[Skylion007]

Also should be noexcept?

[Skylion007]

Suggested change

	memset(&cfg, 0, sizeof(cfg));
	std::memset(&cfg, 0, sizeof(cfg));

[Skylion007]

I forget, do we have a macro for nodiscard?

[henryiii]

git grep nodiscard doesn't reveal anything. That should be in the macro def, so no.