feat: Embeded sub-interpreters (#5666)

* First draft a subinterpreter embedding API

* Move subinterpreter tests to their own file

* Migrate subinterpreter tests to use the new embedded class.

* Add a test for moving subinterpreters across threads for destruction

And find a better way to make that work.

* Code organization

* Add a test which shows demostrates how gil_scoped interacts with sub-interpreters

* Add documentation for embeded sub-interpreters

* Some additional docs work

* Add some convenience accessors

* Add some docs cross references

* Sync some things that were split out into #5665

* Update subinterpreter docs example to not use the CPython api

* Fix pip test

* style: pre-commit fixes

* Fix MSVC warnings

I am surprised other compilers allowed this code with a deleted move ctor.

* Add some sub-headings to the docs

* Oops, make_unique is C++14 so remove it from the tests.

* I think this fixes the EndInterpreter issues on all versions.

It just has to be ifdef'd because it is slightly broken on 3.12, working well on 3.13, and kind of crashy on 3.14beta.  These two verion ifdefs solve all the issues.

* Add a note about exceptions.

They contain Python object references and acquire the GIL, that means they are a danger with subinterpreters!

* style: pre-commit fixes

* Add try/catch to docs examples to match the tips

* Python 3.12 is very picky about this first PyThreadState

Try special casing the destruction on the same thread.

* style: pre-commit fixes

* Missed a rename in a ifdef block

* I think this test is causing problems in 3.12, so try ifdefing it to see if the problems go away.

* style: pre-commit fixes

* Document the 3.12 constraints with a warning

* Apply suggestions from code review

Co-authored-by: Copilot <[email protected]>

* ci: add cpptest to the clang-tidy job

Signed-off-by: Henry Schreiner <[email protected]>

* noexcept move operations

* Update include/pybind11/subinterpreter.h

std::memset

Co-authored-by: Aaron Gokaslan <[email protected]>

---------

Signed-off-by: Henry Schreiner <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Aaron Gokaslan <[email protected]>

Author: 5 people

.github/workflows/format.yml

@@ -55,3 +55,6 @@ jobs:
 
     - name: Build
       run: cmake --build build -j 2 -- --keep-going
+
+    - name: Embedded
+      run: cmake --build build -t cpptest -j 2 -- --keep-going

CMakeLists.txt

@@ -223,6 +223,7 @@ set(PYBIND11_HEADERS
     include/pybind11/operators.h
     include/pybind11/pybind11.h
     include/pybind11/pytypes.h
+    include/pybind11/subinterpreter.h
     include/pybind11/stl.h
     include/pybind11/stl_bind.h
     include/pybind11/stl/filesystem.h

docs/advanced/embedding.rst

@@ -237,31 +237,259 @@ global data. All the details can be found in the CPython documentation.
 
     Creating two concurrent ``scoped_interpreter`` guards is a fatal error. So is
     calling ``initialize_interpreter`` for a second time after the interpreter
-    has already been initialized.
+    has already been initialized. Use :class:`scoped_subinterpreter` to create
+    a sub-interpreter.  See :ref:`subinterp` for important details on sub-interpreters.
 
     Do not use the raw CPython API functions ``Py_Initialize`` and
     ``Py_Finalize`` as these do not properly handle the lifetime of
     pybind11's internal data.
 
 
-Sub-interpreter support
-=======================
+.. _subinterp:
+
+Embedding Sub-interpreters
+==========================
+
+A sub-interpreter is a separate interpreter instance which provides a
+separate, isolated interpreter environment within the same process as the main
+interpreter.  Sub-interpreters are created and managed with a separate API from
+the main interpreter. Beginning in Python 3.12, sub-interpreters each have
+their own Global Interpreter Lock (GIL), which means that running a
+sub-interpreter in a separate thread from the main interpreter can achieve true
+concurrency.
+
+pybind11's sub-interpreter API can be found in ``pybind11/subinterpreter.h``.
+
+pybind11 :class:`subinterpreter` instances can be safely moved and shared between
+threads as needed. However, managing multiple threads and the lifetimes of multiple
+interpreters and their GILs can be challenging.
+Proceed with caution (and lots of testing)!
+
+The main interpreter must be initialized before creating a sub-interpreter, and
+the main interpreter must outlive all sub-interpreters. Sub-interpreters are
+managed through a different API than the main interpreter.
+
+The :class:`subinterpreter` class manages the lifetime of sub-interpreters.
+Instances are movable, but not copyable. Default constructing this class does
+*not* create a sub-interpreter (it creates an empty holder).  To create a
+sub-interpreter, call :func:`subinterpreter::create()`.
+
+.. warning::
+
+    Sub-interpreter creation acquires (and subsequently releases) the main
+    interpreter GIL. If another thread holds the main GIL, the function will
+    block until the main GIL can be acquired.
+
+    Sub-interpreter destruction temporarily activates the sub-interpreter. The
+    sub-interpreter must not be active (on any threads) at the time the
+    :class:`subinterpreter` destructor is called.
+
+    Both actions will re-acquire any interpreter's GIL that was held prior to
+    the call before returning (or return to no active interpreter if none was
+    active at the time of the call).
+
+Each sub-interpreter will import a separate copy of each ``PYBIND11_EMBEDDED_MODULE``
+when those modules specify a ``multiple_interpreters`` tag. If a module does not
+specify a ``multiple_interpreters`` tag, then Python will report an ``ImportError``
+if it is imported in a sub-interpreter.
+
+pybind11 also has a :class:`scoped_subinterpreter` class, which creates and
+activates a sub-interpreter when it is constructed, and deactivates and deletes
+it when it goes out of scope.
+
+Activating a Sub-interpreter
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Once a sub-interpreter is created, you can "activate" it on a thread (and
+acquire its GIL) by creating a :class:`subinterpreter_scoped_activate`
+instance and passing it the sub-intepreter to be activated.  The function
+will acquire the sub-interpreter's GIL and make the sub-interpreter the
+current active interpreter on the current thread for the lifetime of the
+instance. When the :class:`subinterpreter_scoped_activate` instance goes out
+of scope, the sub-interpreter GIL is released and the prior interpreter that
+was active on the thread (if any) is reactivated and it's GIL is re-acquired.
+
+When using ``subinterpreter_scoped_activate``:
+
+1. If the thread holds any interpreter's GIL:
+   - That GIL is released
+2. The new sub-interpreter's GIL is acquired
+3. The new sub-interpreter is made active.
+4. When the scope ends:
+    - The sub-interpreter's GIL is released
+    - If there was a previous interpreter:
+        - The old interpreter's GIL is re-acquired
+        - The old interpreter is made active
+    - Otherwise, no interpreter is currently active and no GIL is held.
+
+Example:
+
+.. code-block:: cpp
+
+    py::initialize_interpreter();
+    // Main GIL is held
+    {
+        py::subinterpreter sub = py::subinterpreter::create();
+        // Main interpreter is still active, main GIL re-acquired
+        {
+            py::subinterpreter_scoped_activate guard(sub);
+            // Sub-interpreter active, thread holds sub's GIL
+            {
+                py::subinterpreter_scoped_activate main_guard(py);
+                // Sub's GIL was automatically released
+                // Main interpreter active, thread holds main's GIL
+            }
+            // Back to sub-interpreter, thread holds sub's GIL again
+        }
+        // Main interpreter is active, main's GIL is held
+    }
+
+
+GIL API for sub-interpreters
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be used to
+manage the GIL of a sub-interpreter just as they do for the main interpreter.
+They both manage the GIL of the currently active interpreter, without the
+programmer having to do anything special or different. There is one important
+caveat:
+
+.. note::
+
+    When no interpreter is active through a
+    :class:`subinterpreter_scoped_activate` instance (such as on a new thread),
+    :class:`gil_scoped_acquire` will acquire the **main** GIL and
+    activate the **main** interpreter.
+
+
+Full Sub-interpreter example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here is an example showing how to create and activate sub-interpreters:
+
+.. code-block:: cpp
+
+    #include <iostream>
+    #include <pybind11/embed.h>
+    #include <pybind11/subinterpreter.h>
+
+    namespace py = pybind11;
+
+    PYBIND11_EMBEDDED_MODULE(printer, m, py::multiple_interpreters::per_interpreter_gil()) {
+        m.def("which", [](const std::string& when) {
+            std::cout << when << "; Current Interpreter is "
+                    << py::subinterpreter::current().id()
+                    << std::endl;
+        });
+    }
+
+    int main() {
+        py::scoped_interpreter main_interp;
+
+        py::module_::import("printer").attr("which")("First init");
+
+        {
+            py::subinterpreter sub = py::subinterpreter::create();
+
+            py::module_::import("printer").attr("which")("Created sub");
+
+            {
+                py::subinterpreter_scoped_activate guard(sub);
+                try {
+                    py::module_::import("printer").attr("which")("Activated sub");
+                }
+                catch (py::error_already_set &e) {
+                    std::cerr << "EXCEPTION " << e.what() << std::endl;
+                    return 1;
+                }
+            }
+
+            py::module_::import("printer").attr("which")("Deactivated sub");
+
+            {
+                py::gil_scoped_release nogil;
+                {
+                    py::subinterpreter_scoped_activate guard(sub);
+                    try {
+                        {
+                            py::subinterpreter_scoped_activate main_guard(py::subinterpreter::main());
+                            try {
+                                py::module_::import("printer").attr("which")("Main within sub");
+                            }
+                            catch (py::error_already_set &e) {
+                                std::cerr << "EXCEPTION " << e.what() << std::endl;
+                                return 1;
+                            }
+                        }
+                        py::module_::import("printer").attr("which")("After Main, still within sub");
+                    }
+                    catch (py::error_already_set &e) {
+                        std::cerr << "EXCEPTION " << e.what() << std::endl;
+                        return 1;
+                    }
+                }
+            }
+        }
+
+        py::module_::import("printer").attr("which")("At end");
+
+        return 0;
+    }
+
+Expected output:
+
+.. code-block:: text
+
+    First init; Current Interpreter is 0
+    Created sub; Current Interpreter is 0
+    Activated sub; Current Interpreter is 1
+    Deactivated sub; Current Interpreter is 0
+    Main within sub; Current Interpreter is 0
+    After Main, still within sub; Current Interpreter is 1
+    At end; Current Interpreter is 0
+
+.. warning::
+
+    In Python 3.12 sub-interpreters must be destroyed in the same OS thread
+    that created them.  Failure to follow this rule may result in deadlocks
+    or crashes when destroying the sub-interpreter on the wrong thread.
+
+    This constraint is not present in Python 3.13+.
+
+
+Best Practices for sub-interpreter safety
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Never share Python objects across different interpreters.
+
+- :class:`error_already_set` objects contain a reference to the Python exception type,
+  and :func:`error_already_set::what()` acquires the GIL. So Python exceptions must
+  **never** be allowed to propagate past the enclosing
+  :class:`subinterpreter_scoped_activate` instance!
+  (So your try/catch should be *just inside* the scope covered by the
+  :class:`subinterpreter_scoped_activate`.)
+
+- Avoid global/static state whenever possible. Instead, keep state within each interpreter,
+  such as within the interpreter state dict, which can be accessed via
+  ``subinterpreter::current().state_dict()``, or within instance members and tied to
+  Python objects.
+
+- Avoid trying to "cache" Python objects in C++ variables across function calls (this is an easy
+  way to accidentally introduce sub-interpreter bugs). In the code example above, note that we
+  did not save the result of :func:`module_::import`, in order to avoid accidentally using the
+  resulting Python object when the wrong interpreter was active.
 
-Creating multiple copies of ``scoped_interpreter`` is not possible because it
-represents the main Python interpreter. Sub-interpreters are something different
-and they do permit the existence of multiple interpreters. This is an advanced
-feature of the CPython API and should be handled with care. pybind11 does not
-currently offer a C++ interface for sub-interpreters, so refer to the CPython
-documentation for all the details regarding this feature.
+- Avoid moving or disarming RAII objects managing GIL and sub-interpreter lifetimes. Doing so can
+  lead to confusion about lifetimes.  (For example, accidentally extending a
+  :class:`subinterpreter_scoped_activate` past the lifetime of it's :class:`subinterpreter`.)
 
-We'll just mention a couple of caveats the sub-interpreters support in pybind11:
+- While sub-interpreters each have their own GIL, there can now be multiple independent GILs in one
+  program so you need to consider the possibility of deadlocks caused by multiple GILs and/or the
+  interactions of the GIL(s) and your C++ code's own locking.
 
- 1. Sub-interpreters will not receive independent copies of embedded modules.
-    Instead, these are shared and modifications in one interpreter may be
-    reflected in another.
+- When using multiple threads to run independent sub-interpreters, the independent GILs allow
+  concurrent calls from different interpreters into the same C++ code from different threads.
+  So you must still consider the thread safety of your C++ code.  Remember, in Python 3.12
+  sub-interpreters must be destroyed on the same thread that they were created on.
 
- 2. Managing multiple threads, multiple interpreters and the GIL can be
-    challenging and there are several caveats here, even within the pure
-    CPython API (please refer to the Python docs for details). As for
-    pybind11, keep in mind that ``gil_scoped_release`` and ``gil_scoped_acquire``
-    do not take sub-interpreters into account.
+- Familiarize yourself with :ref:`misc_concurrency`.

docs/advanced/misc.rst

@@ -228,6 +228,8 @@ You can explicitly disable sub-interpreter support in your module by using the
 :func:`multiple_interpreters::not_supported()` tag. This is the default behavior if you do not
 specify a multiple_interpreters tag.
 
+.. _misc_concurrency:
+
 Concurrency and Parallelism in Python with pybind11
 ===================================================
 

include/pybind11/gil.h

@@ -130,7 +130,7 @@ class gil_scoped_acquire {
     }
 
     /// This method will disable the PyThreadState_DeleteCurrent call and the
-    /// GIL won't be acquired. This method should be used if the interpreter
+    /// GIL won't be released. This method should be used if the interpreter
     /// could be shutting down when this is called, as thread deletion is not
     /// allowed during shutdown. Check _Py_IsFinalizing() on Python 3.7+, and
     /// protect subsequent code.