diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 3d6d1e39fd..bc2db70ea2 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -55,3 +55,6 @@ jobs: - name: Build run: cmake --build build -j 2 -- --keep-going + + - name: Embedded + run: cmake --build build -t cpptest -j 2 -- --keep-going diff --git a/CMakeLists.txt b/CMakeLists.txt index 676fc4b66c..347f816354 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -223,6 +223,7 @@ set(PYBIND11_HEADERS include/pybind11/operators.h include/pybind11/pybind11.h include/pybind11/pytypes.h + include/pybind11/subinterpreter.h include/pybind11/stl.h include/pybind11/stl_bind.h include/pybind11/stl/filesystem.h diff --git a/docs/advanced/embedding.rst b/docs/advanced/embedding.rst index dec767aac9..3ac0579385 100644 --- a/docs/advanced/embedding.rst +++ b/docs/advanced/embedding.rst @@ -237,31 +237,259 @@ global data. All the details can be found in the CPython documentation. Creating two concurrent ``scoped_interpreter`` guards is a fatal error. So is calling ``initialize_interpreter`` for a second time after the interpreter - has already been initialized. + has already been initialized. Use :class:`scoped_subinterpreter` to create + a sub-interpreter. See :ref:`subinterp` for important details on sub-interpreters. Do not use the raw CPython API functions ``Py_Initialize`` and ``Py_Finalize`` as these do not properly handle the lifetime of pybind11's internal data. -Sub-interpreter support -======================= +.. _subinterp: + +Embedding Sub-interpreters +========================== + +A sub-interpreter is a separate interpreter instance which provides a +separate, isolated interpreter environment within the same process as the main +interpreter. Sub-interpreters are created and managed with a separate API from +the main interpreter. Beginning in Python 3.12, sub-interpreters each have +their own Global Interpreter Lock (GIL), which means that running a +sub-interpreter in a separate thread from the main interpreter can achieve true +concurrency. + +pybind11's sub-interpreter API can be found in ``pybind11/subinterpreter.h``. + +pybind11 :class:`subinterpreter` instances can be safely moved and shared between +threads as needed. However, managing multiple threads and the lifetimes of multiple +interpreters and their GILs can be challenging. +Proceed with caution (and lots of testing)! + +The main interpreter must be initialized before creating a sub-interpreter, and +the main interpreter must outlive all sub-interpreters. Sub-interpreters are +managed through a different API than the main interpreter. + +The :class:`subinterpreter` class manages the lifetime of sub-interpreters. +Instances are movable, but not copyable. Default constructing this class does +*not* create a sub-interpreter (it creates an empty holder). To create a +sub-interpreter, call :func:`subinterpreter::create()`. + +.. warning:: + + Sub-interpreter creation acquires (and subsequently releases) the main + interpreter GIL. If another thread holds the main GIL, the function will + block until the main GIL can be acquired. + + Sub-interpreter destruction temporarily activates the sub-interpreter. The + sub-interpreter must not be active (on any threads) at the time the + :class:`subinterpreter` destructor is called. + + Both actions will re-acquire any interpreter's GIL that was held prior to + the call before returning (or return to no active interpreter if none was + active at the time of the call). + +Each sub-interpreter will import a separate copy of each ``PYBIND11_EMBEDDED_MODULE`` +when those modules specify a ``multiple_interpreters`` tag. If a module does not +specify a ``multiple_interpreters`` tag, then Python will report an ``ImportError`` +if it is imported in a sub-interpreter. + +pybind11 also has a :class:`scoped_subinterpreter` class, which creates and +activates a sub-interpreter when it is constructed, and deactivates and deletes +it when it goes out of scope. + +Activating a Sub-interpreter +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Once a sub-interpreter is created, you can "activate" it on a thread (and +acquire its GIL) by creating a :class:`subinterpreter_scoped_activate` +instance and passing it the sub-intepreter to be activated. The function +will acquire the sub-interpreter's GIL and make the sub-interpreter the +current active interpreter on the current thread for the lifetime of the +instance. When the :class:`subinterpreter_scoped_activate` instance goes out +of scope, the sub-interpreter GIL is released and the prior interpreter that +was active on the thread (if any) is reactivated and it's GIL is re-acquired. + +When using ``subinterpreter_scoped_activate``: + +1. If the thread holds any interpreter's GIL: + - That GIL is released +2. The new sub-interpreter's GIL is acquired +3. The new sub-interpreter is made active. +4. When the scope ends: + - The sub-interpreter's GIL is released + - If there was a previous interpreter: + - The old interpreter's GIL is re-acquired + - The old interpreter is made active + - Otherwise, no interpreter is currently active and no GIL is held. + +Example: + +.. code-block:: cpp + + py::initialize_interpreter(); + // Main GIL is held + { + py::subinterpreter sub = py::subinterpreter::create(); + // Main interpreter is still active, main GIL re-acquired + { + py::subinterpreter_scoped_activate guard(sub); + // Sub-interpreter active, thread holds sub's GIL + { + py::subinterpreter_scoped_activate main_guard(py); + // Sub's GIL was automatically released + // Main interpreter active, thread holds main's GIL + } + // Back to sub-interpreter, thread holds sub's GIL again + } + // Main interpreter is active, main's GIL is held + } + + +GIL API for sub-interpreters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +:class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be used to +manage the GIL of a sub-interpreter just as they do for the main interpreter. +They both manage the GIL of the currently active interpreter, without the +programmer having to do anything special or different. There is one important +caveat: + +.. note:: + + When no interpreter is active through a + :class:`subinterpreter_scoped_activate` instance (such as on a new thread), + :class:`gil_scoped_acquire` will acquire the **main** GIL and + activate the **main** interpreter. + + +Full Sub-interpreter example +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Here is an example showing how to create and activate sub-interpreters: + +.. code-block:: cpp + + #include + #include + #include + + namespace py = pybind11; + + PYBIND11_EMBEDDED_MODULE(printer, m, py::multiple_interpreters::per_interpreter_gil()) { + m.def("which", [](const std::string& when) { + std::cout << when << "; Current Interpreter is " + << py::subinterpreter::current().id() + << std::endl; + }); + } + + int main() { + py::scoped_interpreter main_interp; + + py::module_::import("printer").attr("which")("First init"); + + { + py::subinterpreter sub = py::subinterpreter::create(); + + py::module_::import("printer").attr("which")("Created sub"); + + { + py::subinterpreter_scoped_activate guard(sub); + try { + py::module_::import("printer").attr("which")("Activated sub"); + } + catch (py::error_already_set &e) { + std::cerr << "EXCEPTION " << e.what() << std::endl; + return 1; + } + } + + py::module_::import("printer").attr("which")("Deactivated sub"); + + { + py::gil_scoped_release nogil; + { + py::subinterpreter_scoped_activate guard(sub); + try { + { + py::subinterpreter_scoped_activate main_guard(py::subinterpreter::main()); + try { + py::module_::import("printer").attr("which")("Main within sub"); + } + catch (py::error_already_set &e) { + std::cerr << "EXCEPTION " << e.what() << std::endl; + return 1; + } + } + py::module_::import("printer").attr("which")("After Main, still within sub"); + } + catch (py::error_already_set &e) { + std::cerr << "EXCEPTION " << e.what() << std::endl; + return 1; + } + } + } + } + + py::module_::import("printer").attr("which")("At end"); + + return 0; + } + +Expected output: + +.. code-block:: text + + First init; Current Interpreter is 0 + Created sub; Current Interpreter is 0 + Activated sub; Current Interpreter is 1 + Deactivated sub; Current Interpreter is 0 + Main within sub; Current Interpreter is 0 + After Main, still within sub; Current Interpreter is 1 + At end; Current Interpreter is 0 + +.. warning:: + + In Python 3.12 sub-interpreters must be destroyed in the same OS thread + that created them. Failure to follow this rule may result in deadlocks + or crashes when destroying the sub-interpreter on the wrong thread. + + This constraint is not present in Python 3.13+. + + +Best Practices for sub-interpreter safety +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Never share Python objects across different interpreters. + +- :class:`error_already_set` objects contain a reference to the Python exception type, + and :func:`error_already_set::what()` acquires the GIL. So Python exceptions must + **never** be allowed to propagate past the enclosing + :class:`subinterpreter_scoped_activate` instance! + (So your try/catch should be *just inside* the scope covered by the + :class:`subinterpreter_scoped_activate`.) + +- Avoid global/static state whenever possible. Instead, keep state within each interpreter, + such as within the interpreter state dict, which can be accessed via + ``subinterpreter::current().state_dict()``, or within instance members and tied to + Python objects. + +- Avoid trying to "cache" Python objects in C++ variables across function calls (this is an easy + way to accidentally introduce sub-interpreter bugs). In the code example above, note that we + did not save the result of :func:`module_::import`, in order to avoid accidentally using the + resulting Python object when the wrong interpreter was active. -Creating multiple copies of ``scoped_interpreter`` is not possible because it -represents the main Python interpreter. Sub-interpreters are something different -and they do permit the existence of multiple interpreters. This is an advanced -feature of the CPython API and should be handled with care. pybind11 does not -currently offer a C++ interface for sub-interpreters, so refer to the CPython -documentation for all the details regarding this feature. +- Avoid moving or disarming RAII objects managing GIL and sub-interpreter lifetimes. Doing so can + lead to confusion about lifetimes. (For example, accidentally extending a + :class:`subinterpreter_scoped_activate` past the lifetime of it's :class:`subinterpreter`.) -We'll just mention a couple of caveats the sub-interpreters support in pybind11: +- While sub-interpreters each have their own GIL, there can now be multiple independent GILs in one + program so you need to consider the possibility of deadlocks caused by multiple GILs and/or the + interactions of the GIL(s) and your C++ code's own locking. - 1. Sub-interpreters will not receive independent copies of embedded modules. - Instead, these are shared and modifications in one interpreter may be - reflected in another. +- When using multiple threads to run independent sub-interpreters, the independent GILs allow + concurrent calls from different interpreters into the same C++ code from different threads. + So you must still consider the thread safety of your C++ code. Remember, in Python 3.12 + sub-interpreters must be destroyed on the same thread that they were created on. - 2. Managing multiple threads, multiple interpreters and the GIL can be - challenging and there are several caveats here, even within the pure - CPython API (please refer to the Python docs for details). As for - pybind11, keep in mind that ``gil_scoped_release`` and ``gil_scoped_acquire`` - do not take sub-interpreters into account. +- Familiarize yourself with :ref:`misc_concurrency`. diff --git a/docs/advanced/misc.rst b/docs/advanced/misc.rst index 7d2a279585..b8cb1923e9 100644 --- a/docs/advanced/misc.rst +++ b/docs/advanced/misc.rst @@ -228,6 +228,8 @@ You can explicitly disable sub-interpreter support in your module by using the :func:`multiple_interpreters::not_supported()` tag. This is the default behavior if you do not specify a multiple_interpreters tag. +.. _misc_concurrency: + Concurrency and Parallelism in Python with pybind11 =================================================== diff --git a/include/pybind11/gil.h b/include/pybind11/gil.h index 1a9bfeaddf..e90ea41528 100644 --- a/include/pybind11/gil.h +++ b/include/pybind11/gil.h @@ -130,7 +130,7 @@ class gil_scoped_acquire { } /// This method will disable the PyThreadState_DeleteCurrent call and the - /// GIL won't be acquired. This method should be used if the interpreter + /// GIL won't be released. This method should be used if the interpreter /// could be shutting down when this is called, as thread deletion is not /// allowed during shutdown. Check _Py_IsFinalizing() on Python 3.7+, and /// protect subsequent code. diff --git a/include/pybind11/subinterpreter.h b/include/pybind11/subinterpreter.h new file mode 100644 index 0000000000..306e47bc6a --- /dev/null +++ b/include/pybind11/subinterpreter.h @@ -0,0 +1,320 @@ +/* + pybind11/subinterpreter.h: Support for creating and using subinterpreters + + Copyright (c) 2025 The Pybind Development Team. + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +#pragma once + +#include "detail/common.h" +#include "detail/internals.h" +#include "gil.h" + +#include + +#if !defined(PYBIND11_SUBINTERPRETER_SUPPORT) +# error "This platform does not support subinterpreters, do not include this file." +#endif + +PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_NAMESPACE_BEGIN(detail) +PyInterpreterState *get_interpreter_state_unchecked() { + auto cur_tstate = get_thread_state_unchecked(); + if (cur_tstate) + return cur_tstate->interp; + else + return nullptr; +} +PYBIND11_NAMESPACE_END(detail) + +class subinterpreter; + +/// Activate the subinterpreter and acquire its GIL, while also releasing any GIL and interpreter +/// currently held. Upon exiting the scope, the previous subinterpreter (if any) and its +/// associated GIL are restored to their state as they were before the scope was entered. +class subinterpreter_scoped_activate { +public: + explicit subinterpreter_scoped_activate(subinterpreter const &si); + ~subinterpreter_scoped_activate(); + + subinterpreter_scoped_activate(subinterpreter_scoped_activate &&) = delete; + subinterpreter_scoped_activate(subinterpreter_scoped_activate const &) = delete; + subinterpreter_scoped_activate &operator=(subinterpreter_scoped_activate &) = delete; + subinterpreter_scoped_activate &operator=(subinterpreter_scoped_activate const &) = delete; + +private: + PyThreadState *old_tstate_ = nullptr; + PyThreadState *tstate_ = nullptr; + PyGILState_STATE gil_state_; + bool simple_gil_ = false; +}; + +/// Holds a Python subinterpreter instance +class subinterpreter { +public: + /// empty/unusable, but move-assignable. use create() to create a subinterpreter. + subinterpreter() = default; + + subinterpreter(subinterpreter const ©) = delete; + subinterpreter &operator=(subinterpreter const ©) = delete; + + subinterpreter(subinterpreter &&old) noexcept + : istate_(old.istate_), creation_tstate_(old.creation_tstate_) { + old.istate_ = nullptr; + old.creation_tstate_ = nullptr; + } + + subinterpreter &operator=(subinterpreter &&old) noexcept { + std::swap(old.istate_, istate_); + std::swap(old.creation_tstate_, creation_tstate_); + return *this; + } + + /// Create a new subinterpreter with the specified configuration + /// @note This function acquires (and then releases) the main interpreter GIL, but the main + /// interpreter and its GIL are not required to be held prior to calling this function. + static inline subinterpreter create(PyInterpreterConfig const &cfg) { + error_scope err_scope; + subinterpreter result; + { + // we must hold the main GIL in order to create a subinterpreter + subinterpreter_scoped_activate main_guard(main()); + + auto prev_tstate = PyThreadState_Get(); + + auto status = Py_NewInterpreterFromConfig(&result.creation_tstate_, &cfg); + + // this doesn't raise a normal Python exception, it provides an exit() status code. + if (PyStatus_Exception(status)) { + pybind11_fail("failed to create new sub-interpreter"); + } + + // upon success, the new interpreter is activated in this thread + result.istate_ = result.creation_tstate_->interp; + detail::get_num_interpreters_seen() += 1; // there are now many interpreters + detail::get_internals(); // initialize internals.tstate, amongst other things... + + // In 3.13+ this state should be deleted right away, and the memory will be reused for + // the next threadstate on this interpreter. However, on 3.12 we cannot do that, we + // must keep it around (but not use it) ... see destructor. +#if PY_VERSION_HEX >= 0x030D0000 + PyThreadState_Clear(result.creation_tstate_); + PyThreadState_DeleteCurrent(); +#endif + + // we have to switch back to main, and then the scopes will handle cleanup + PyThreadState_Swap(prev_tstate); + } + return result; + } + + /// Calls create() with a default configuration of an isolated interpreter that disallows fork, + /// exec, and Python threads. + static inline subinterpreter create() { + // same as the default config in the python docs + PyInterpreterConfig cfg; + std::memset(&cfg, 0, sizeof(cfg)); + cfg.check_multi_interp_extensions = 1; + cfg.gil = PyInterpreterConfig_OWN_GIL; + return create(cfg); + } + + ~subinterpreter() { + if (!creation_tstate_) { + // non-owning wrapper, do nothing. + return; + } + + PyThreadState *destroy_tstate; + PyThreadState *old_tstate; + + // Python 3.12 requires us to keep the original PyThreadState alive until we are ready to + // destroy the interpreter. We prefer to use that to destroy the interpreter. +#if PY_VERSION_HEX < 0x030D0000 + // The tstate passed to Py_EndInterpreter MUST have been created on the current OS thread. + bool same_thread = false; +# ifdef PY_HAVE_THREAD_NATIVE_ID + same_thread = PyThread_get_thread_native_id() == creation_tstate_->native_thread_id; +# endif + if (same_thread) { + // OK it is safe to use the creation state here + destroy_tstate = creation_tstate_; + old_tstate = PyThreadState_Swap(destroy_tstate); + } else { + // We have to make a new tstate on this thread and use that. + destroy_tstate = PyThreadState_New(istate_); + old_tstate = PyThreadState_Swap(destroy_tstate); + + // We can use the one we just created, so we must delete the creation state. + PyThreadState_Clear(creation_tstate_); + PyThreadState_Delete(creation_tstate_); + } +#else + destroy_tstate = PyThreadState_New(istate_); + old_tstate = PyThreadState_Swap(destroy_tstate); +#endif + + bool switch_back = old_tstate && old_tstate->interp != istate_; + + // Get the internals pointer (without creating it if it doesn't exist). It's possible + // for the internals to be created during Py_EndInterpreter() (e.g. if a py::capsule + // calls `get_internals()` during destruction), so we get the pointer-pointer here and + // check it after. + auto *&internals_ptr_ptr = detail::get_internals_pp(); + auto *&local_internals_ptr_ptr = detail::get_internals_pp(); + { + dict sd = state_dict(); + internals_ptr_ptr + = detail::get_internals_pp_from_capsule_in_state_dict( + sd, PYBIND11_INTERNALS_ID); + local_internals_ptr_ptr + = detail::get_internals_pp_from_capsule_in_state_dict( + sd, detail::get_local_internals_id()); + } + + // End it + Py_EndInterpreter(destroy_tstate); + + // do NOT decrease detail::get_num_interpreters_seen, because it can never decrease + // while other threads are running... + + if (internals_ptr_ptr) { + internals_ptr_ptr->reset(); + } + if (local_internals_ptr_ptr) { + local_internals_ptr_ptr->reset(); + } + + // switch back to the old tstate and old GIL (if there was one) + if (switch_back) + PyThreadState_Swap(old_tstate); + } + + /// Get a handle to the main interpreter that can be used with subinterpreter_scoped_activate + /// Note that destructing the handle is a noop, the main interpreter can only be ended by + /// py::finalize_interpreter() + static subinterpreter main() { + subinterpreter m; + m.istate_ = PyInterpreterState_Main(); + m.disarm(); // make destruct a noop + return m; + } + + /// Get a non-owning wrapper of the currently active interpreter (if any) + static subinterpreter current() { + subinterpreter c; + c.istate_ = detail::get_interpreter_state_unchecked(); + c.disarm(); // make destruct a noop, we don't own this... + return c; + } + + /// Get the numerical identifier for the sub-interpreter + int64_t id() const { + if (istate_ != nullptr) + return PyInterpreterState_GetID(istate_); + else + return -1; // CPython uses one-up numbers from 0, so negative should be safe to return + // here. + } + + /// Get the interpreter's state dict. This interpreter's GIL must be held before calling! + dict state_dict() { return reinterpret_borrow(PyInterpreterState_GetDict(istate_)); } + + /// abandon cleanup of this subinterpreter (leak it). this might be needed during + /// finalization... + void disarm() { creation_tstate_ = nullptr; } + + /// An empty wrapper cannot be activated + bool empty() const { return istate_ == nullptr; } + + /// Is this wrapper non-empty + explicit operator bool() const { return !empty(); } + +private: + friend class subinterpreter_scoped_activate; + PyInterpreterState *istate_ = nullptr; + PyThreadState *creation_tstate_ = nullptr; +}; + +class scoped_subinterpreter { +public: + scoped_subinterpreter() : si_(subinterpreter::create()), scope_(si_) {} + + explicit scoped_subinterpreter(PyInterpreterConfig const &cfg) + : si_(subinterpreter::create(cfg)), scope_(si_) {} + +private: + subinterpreter si_; + subinterpreter_scoped_activate scope_; +}; + +inline subinterpreter_scoped_activate::subinterpreter_scoped_activate(subinterpreter const &si) { + if (!si.istate_) { + pybind11_fail("null subinterpreter"); + } + + if (detail::get_interpreter_state_unchecked() == si.istate_) { + // we are already on this interpreter, make sure we hold the GIL + simple_gil_ = true; + gil_state_ = PyGILState_Ensure(); + return; + } + + // we can't really interact with the interpreter at all until we switch to it + // not even to, for example, look in its state dict or touch its internals + tstate_ = PyThreadState_New(si.istate_); + + // make the interpreter active and acquire the GIL + old_tstate_ = PyThreadState_Swap(tstate_); + + // save this in internals for scoped_gil calls + PYBIND11_TLS_REPLACE_VALUE(detail::get_internals().tstate, tstate_); +} + +inline subinterpreter_scoped_activate::~subinterpreter_scoped_activate() { + if (simple_gil_) { + // We were on this interpreter already, so just make sure the GIL goes back as it was + PyGILState_Release(gil_state_); + } else { +#if defined(PYBIND11_DETAILED_ERROR_MESSAGES) + bool has_active_exception; +# if defined(__cpp_lib_uncaught_exceptions) + has_active_exception = std::uncaught_exceptions() > 0; +# else + // removed in C++20, replaced with uncaught_exceptions + has_active_exception = std::uncaught_exception(); +# endif + if (has_active_exception) { + try { + std::rethrow_exception(std::current_exception()); + } catch (error_already_set &) { + // Because error_already_set holds python objects and what() acquires the GIL, it + // is basically never OK to let these exceptions propagate outside the current + // active interpreter. + pybind11_fail("~subinterpreter_scoped_activate: cannot propagate Python " + "exceptions outside of their owning interpreter"); + } catch (...) { + } + } +#endif + + if (tstate_) { +#if defined(PYBIND11_DETAILED_ERROR_MESSAGES) + if (detail::get_thread_state_unchecked() != tstate_) { + pybind11_fail("~subinterpreter_scoped_activate: thread state must be current!"); + } +#endif + PYBIND11_TLS_DELETE_VALUE(detail::get_internals().tstate); + PyThreadState_Clear(tstate_); + PyThreadState_DeleteCurrent(); + } + + // Go back the previous interpreter (if any) and acquire THAT gil + PyThreadState_Swap(old_tstate_); + } +} + +PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/tests/extra_python_package/test_files.py b/tests/extra_python_package/test_files.py index 2919cf3ea3..9725bedae1 100644 --- a/tests/extra_python_package/test_files.py +++ b/tests/extra_python_package/test_files.py @@ -58,6 +58,7 @@ "include/pybind11/options.h", "include/pybind11/pybind11.h", "include/pybind11/pytypes.h", + "include/pybind11/subinterpreter.h", "include/pybind11/stl.h", "include/pybind11/stl_bind.h", "include/pybind11/trampoline_self_life_support.h", diff --git a/tests/test_embed/CMakeLists.txt b/tests/test_embed/CMakeLists.txt index 1a537a6580..af3c848ad4 100644 --- a/tests/test_embed/CMakeLists.txt +++ b/tests/test_embed/CMakeLists.txt @@ -28,7 +28,7 @@ endif() find_package(Threads REQUIRED) -add_executable(test_embed catch.cpp test_interpreter.cpp) +add_executable(test_embed catch.cpp test_interpreter.cpp test_subinterpreter.cpp) pybind11_enable_warnings(test_embed) target_link_libraries(test_embed PRIVATE pybind11::embed Catch2::Catch2 Threads::Threads) diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 6e4be7378a..e555c0d70c 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -19,6 +19,21 @@ size_t get_sys_path_size() { return py::len(sys_path); } +bool has_state_dict_internals_obj() { + py::dict state = py::detail::get_python_state_dict(); + return state.contains(PYBIND11_INTERNALS_ID); +} + +bool has_pybind11_internals_static() { + auto *&ipp = py::detail::get_internals_pp(); + return (ipp != nullptr) && *ipp; +} + +uintptr_t get_details_as_uintptr() { + return reinterpret_cast( + py::detail::get_internals_pp()->get()); +} + class Widget { public: explicit Widget(std::string message) : message(std::move(message)) {} @@ -258,21 +273,6 @@ TEST_CASE("Add program dir to path using PyConfig") { } #endif -bool has_state_dict_internals_obj() { - py::dict state = py::detail::get_python_state_dict(); - return state.contains(PYBIND11_INTERNALS_ID); -} - -bool has_pybind11_internals_static() { - auto *&ipp = py::detail::get_internals_pp(); - return (ipp != nullptr) && *ipp; -} - -uintptr_t get_details_as_uintptr() { - return reinterpret_cast( - py::detail::get_internals_pp()->get()); -} - TEST_CASE("Restart the interpreter") { // Verify pre-restart state. REQUIRE(py::module_::import("widget_module").attr("add")(1, 2).cast() == 3); @@ -336,279 +336,6 @@ TEST_CASE("Restart the interpreter") { REQUIRE(py_widget.attr("the_message").cast() == "Hello after restart"); } -#if defined(PYBIND11_SUBINTERPRETER_SUPPORT) -TEST_CASE("Subinterpreter") { - py::module_::import("external_module"); // in the main interpreter - - // Add tags to the modules in the main interpreter and test the basics. - py::module_::import("__main__").attr("main_tag") = "main interpreter"; - { - auto m = py::module_::import("widget_module"); - m.attr("extension_module_tag") = "added to module in main interpreter"; - - REQUIRE(m.attr("add")(1, 2).cast() == 3); - } - - auto main_int - = py::module_::import("external_module").attr("internals_at")().cast(); - - REQUIRE(has_state_dict_internals_obj()); - REQUIRE(has_pybind11_internals_static()); - - /// Create and switch to a subinterpreter. - auto *main_tstate = PyThreadState_Get(); - auto *sub_tstate = Py_NewInterpreter(); - - py::detail::get_num_interpreters_seen()++; - - // Subinterpreters get their own copy of builtins. - REQUIRE_FALSE(has_state_dict_internals_obj()); - - // internals hasn't been populated yet, but will be different for the subinterpreter - REQUIRE_FALSE(has_pybind11_internals_static()); - - py::list(py::module_::import("sys").attr("path")).append(py::str(".")); - - auto ext_int = py::module_::import("external_module").attr("internals_at")().cast(); - py::detail::get_internals(); - REQUIRE(has_pybind11_internals_static()); - REQUIRE(get_details_as_uintptr() == ext_int); - REQUIRE(main_int != ext_int); - - // Modules tags should be gone. - REQUIRE_FALSE(py::hasattr(py::module_::import("__main__"), "tag")); - { - REQUIRE_NOTHROW(py::module_::import("widget_module")); - auto m = py::module_::import("widget_module"); - REQUIRE_FALSE(py::hasattr(m, "extension_module_tag")); - - // Function bindings should still work. - REQUIRE(m.attr("add")(1, 2).cast() == 3); - } - - // The subinterpreter now has internals populated since we imported a pybind11 module - REQUIRE(has_pybind11_internals_static()); - - // Restore main interpreter. - Py_EndInterpreter(sub_tstate); - py::detail::get_num_interpreters_seen() = 1; - PyThreadState_Swap(main_tstate); - - REQUIRE(py::hasattr(py::module_::import("__main__"), "main_tag")); - REQUIRE(py::hasattr(py::module_::import("widget_module"), "extension_module_tag")); - REQUIRE(has_state_dict_internals_obj()); -} - -TEST_CASE("Multiple Subinterpreters") { - // Make sure the module is in the main interpreter and save its pointer - auto *main_ext = py::module_::import("external_module").ptr(); - auto main_int - = py::module_::import("external_module").attr("internals_at")().cast(); - py::module_::import("external_module").attr("multi_interp") = "1"; - - auto *main_tstate = PyThreadState_Get(); - - /// Create and switch to a subinterpreter. - auto *sub1_tstate = Py_NewInterpreter(); - py::detail::get_num_interpreters_seen()++; - - py::list(py::module_::import("sys").attr("path")).append(py::str(".")); - - // The subinterpreter has its own copy of this module which is completely separate from main - auto *sub1_ext = py::module_::import("external_module").ptr(); - REQUIRE(sub1_ext != main_ext); - REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp")); - py::module_::import("external_module").attr("multi_interp") = "2"; - // The subinterpreter also has its own internals - auto sub1_int - = py::module_::import("external_module").attr("internals_at")().cast(); - REQUIRE(sub1_int != main_int); - - // Create another interpreter - auto *sub2_tstate = Py_NewInterpreter(); - py::detail::get_num_interpreters_seen()++; - - py::list(py::module_::import("sys").attr("path")).append(py::str(".")); - - // The second subinterpreter is separate from both main and the other subinterpreter - auto *sub2_ext = py::module_::import("external_module").ptr(); - REQUIRE(sub2_ext != main_ext); - REQUIRE(sub2_ext != sub1_ext); - REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp")); - py::module_::import("external_module").attr("multi_interp") = "3"; - // The subinterpreter also has its own internals - auto sub2_int - = py::module_::import("external_module").attr("internals_at")().cast(); - REQUIRE(sub2_int != main_int); - REQUIRE(sub2_int != sub1_int); - - PyThreadState_Swap(sub1_tstate); // go back to sub1 - - REQUIRE(py::cast(py::module_::import("external_module").attr("multi_interp")) - == "2"); - - PyThreadState_Swap(main_tstate); // go back to main - - auto post_int - = py::module_::import("external_module").attr("internals_at")().cast(); - // Make sure internals went back the way it was before - REQUIRE(main_int == post_int); - - REQUIRE(py::cast(py::module_::import("external_module").attr("multi_interp")) - == "1"); - - PyThreadState_Swap(sub1_tstate); - Py_EndInterpreter(sub1_tstate); - PyThreadState_Swap(sub2_tstate); - Py_EndInterpreter(sub2_tstate); - - py::detail::get_num_interpreters_seen() = 1; - PyThreadState_Swap(main_tstate); -} -#endif - -#if defined(Py_MOD_PER_INTERPRETER_GIL_SUPPORTED) && defined(PYBIND11_SUBINTERPRETER_SUPPORT) -TEST_CASE("Per-Subinterpreter GIL") { - auto main_int - = py::module_::import("external_module").attr("internals_at")().cast(); - - std::atomic started, sync, failure; - started = 0; - sync = 0; - failure = 0; - -// REQUIRE throws on failure, so we can't use it within the thread -# define T_REQUIRE(status) \ - do { \ - assert(status); \ - if (!(status)) \ - ++failure; \ - } while (0) - - auto &&thread_main = [&](int num) { - while (started == 0) - std::this_thread::sleep_for(std::chrono::microseconds(1)); - ++started; - - py::gil_scoped_acquire gil; - auto main_tstate = PyThreadState_Get(); - - // we have the GIL, we can access the main interpreter - auto t_int - = py::module_::import("external_module").attr("internals_at")().cast(); - T_REQUIRE(t_int == main_int); - py::module_::import("external_module").attr("multi_interp") = "1"; - - PyThreadState *sub = nullptr; - PyInterpreterConfig cfg; - memset(&cfg, 0, sizeof(cfg)); - cfg.check_multi_interp_extensions = 1; - cfg.gil = PyInterpreterConfig_OWN_GIL; - auto status = Py_NewInterpreterFromConfig(&sub, &cfg); - T_REQUIRE(!PyStatus_IsError(status)); - - py::detail::get_num_interpreters_seen()++; - - py::list(py::module_::import("sys").attr("path")).append(py::str(".")); - - // we have switched to the new interpreter and released the main gil - - // trampoline_module did not provide the per_interpreter_gil tag, so it cannot be - // imported - bool caught = false; - try { - py::module_::import("trampoline_module"); - } catch (pybind11::error_already_set &pe) { - T_REQUIRE(pe.matches(PyExc_ImportError)); - std::string msg(pe.what()); - T_REQUIRE(msg.find("does not support loading in subinterpreters") - != std::string::npos); - caught = true; - } - T_REQUIRE(caught); - - // widget_module did provide the per_interpreter_gil tag, so it this does not throw - py::module_::import("widget_module"); - - T_REQUIRE(!py::hasattr(py::module_::import("external_module"), "multi_interp")); - py::module_::import("external_module").attr("multi_interp") = std::to_string(num); - - // wait for something to set sync to our thread number - // we are holding our subinterpreter's GIL - while (sync != num) - std::this_thread::sleep_for(std::chrono::microseconds(1)); - - // now change it so the next thread can mvoe on - ++sync; - - // but keep holding the GIL until after the next thread moves on as well - while (sync == num + 1) - std::this_thread::sleep_for(std::chrono::microseconds(1)); - - // one last check before quitting the thread, the internals should be different - auto sub_int - = py::module_::import("external_module").attr("internals_at")().cast(); - T_REQUIRE(sub_int != main_int); - - Py_EndInterpreter(sub); - - // switch back so the scoped_acquire can release the GIL properly - PyThreadState_Swap(main_tstate); - }; - - std::thread t1(thread_main, 1); - std::thread t2(thread_main, 2); - - // we spawned two threads, at this point they are both waiting for started to increase - ++started; - - // ok now wait for the threads to start - while (started != 3) - std::this_thread::sleep_for(std::chrono::microseconds(1)); - - // we still hold the main GIL, at this point both threads are waiting on the main GIL - // IN THE CASE of free threading, the threads are waiting on sync (because there is no GIL) - - // IF the below code hangs in one of the wait loops, then the child thread GIL behavior did not - // function as expected. - { - // release the GIL and allow the threads to run - py::gil_scoped_release nogil; - - // the threads are now waiting on the sync - REQUIRE(sync == 0); - - // this will trigger thread 1 and then advance and trigger 2 and then advance - sync = 1; - - // wait for thread 2 to advance - while (sync != 3) - std::this_thread::sleep_for(std::chrono::microseconds(1)); - - // we know now that thread 1 has run and may be finishing - // and thread 2 is waiting for permission to advance - - // so we move sync so that thread 2 can finish executing - ++sync; - - // now wait for both threads to complete - t1.join(); - t2.join(); - } - - // now we have the gil again, sanity check - REQUIRE(py::cast(py::module_::import("external_module").attr("multi_interp")) - == "1"); - - // the threads are stopped. we can now lower this for the rest of the test - py::detail::get_num_interpreters_seen() = 1; - - // make sure nothing unexpected happened inside the threads, now that they are completed - REQUIRE(failure == 0); -# undef T_REQUIRE -} -#endif - TEST_CASE("Execution frame") { // When the interpreter is embedded, there is no execution frame, but `py::exec` // should still function by using reasonable globals: `__main__.__dict__`. diff --git a/tests/test_embed/test_subinterpreter.cpp b/tests/test_embed/test_subinterpreter.cpp new file mode 100644 index 0000000000..9d7d88b8a0 --- /dev/null +++ b/tests/test_embed/test_subinterpreter.cpp @@ -0,0 +1,431 @@ +#include +#ifdef PYBIND11_SUBINTERPRETER_SUPPORT +# include + +// Silence MSVC C++17 deprecation warning from Catch regarding std::uncaught_exceptions (up to +// catch 2.0.1; this should be fixed in the next catch release after 2.0.1). +PYBIND11_WARNING_DISABLE_MSVC(4996) + +# include +# include +# include +# include +# include +# include + +namespace py = pybind11; +using namespace py::literals; + +bool has_state_dict_internals_obj(); +bool has_pybind11_internals_static(); +uintptr_t get_details_as_uintptr(); + +void unsafe_reset_internals_for_single_interpreter() { + // unsafe normally, but for subsequent tests, put this back.. we know there are no threads + // running and only 1 interpreter + py::detail::get_num_interpreters_seen() = 1; + py::detail::get_internals_pp() = nullptr; + py::detail::get_internals(); + py::detail::get_internals_pp() = nullptr; + py::detail::get_local_internals(); +} + +TEST_CASE("Single Subinterpreter") { + py::module_::import("external_module"); // in the main interpreter + + // Add tags to the modules in the main interpreter and test the basics. + py::module_::import("__main__").attr("main_tag") = "main interpreter"; + { + auto m = py::module_::import("widget_module"); + m.attr("extension_module_tag") = "added to module in main interpreter"; + + REQUIRE(m.attr("add")(1, 2).cast() == 3); + } + REQUIRE(has_state_dict_internals_obj()); + REQUIRE(has_pybind11_internals_static()); + + auto main_int + = py::module_::import("external_module").attr("internals_at")().cast(); + + /// Create and switch to a subinterpreter. + { + py::scoped_subinterpreter ssi; + + // The subinterpreter has internals populated + REQUIRE(has_pybind11_internals_static()); + + py::list(py::module_::import("sys").attr("path")).append(py::str(".")); + + auto ext_int + = py::module_::import("external_module").attr("internals_at")().cast(); + py::detail::get_internals(); + REQUIRE(has_pybind11_internals_static()); + REQUIRE(get_details_as_uintptr() == ext_int); + REQUIRE(ext_int != main_int); + + // Modules tags should be gone. + REQUIRE_FALSE(py::hasattr(py::module_::import("__main__"), "tag")); + { + auto m = py::module_::import("widget_module"); + REQUIRE_FALSE(py::hasattr(m, "extension_module_tag")); + + // Function bindings should still work. + REQUIRE(m.attr("add")(1, 2).cast() == 3); + } + } + + REQUIRE(py::hasattr(py::module_::import("__main__"), "main_tag")); + REQUIRE(py::hasattr(py::module_::import("widget_module"), "extension_module_tag")); + REQUIRE(has_state_dict_internals_obj()); + + unsafe_reset_internals_for_single_interpreter(); +} + +# if PY_VERSION_HEX >= 0x030D0000 +TEST_CASE("Move Subinterpreter") { + std::unique_ptr sub(new py::subinterpreter(py::subinterpreter::create())); + + // on this thread, use the subinterpreter and import some non-trivial junk + { + py::subinterpreter_scoped_activate activate(*sub); + + py::list(py::module_::import("sys").attr("path")).append(py::str(".")); + py::module_::import("datetime"); + py::module_::import("threading"); + py::module_::import("external_module"); + } + + std::thread([&]() { + // Use it again + { + py::subinterpreter_scoped_activate activate(*sub); + py::module_::import("external_module"); + } + sub.reset(); + }).join(); + + REQUIRE(!sub); + + unsafe_reset_internals_for_single_interpreter(); +} +# endif + +TEST_CASE("GIL Subinterpreter") { + + PyInterpreterState *main_interp = PyInterpreterState_Get(); + + { + auto sub = py::subinterpreter::create(); + + REQUIRE(main_interp == PyInterpreterState_Get()); + + PyInterpreterState *sub_interp = nullptr; + + { + py::subinterpreter_scoped_activate activate(sub); + + sub_interp = PyInterpreterState_Get(); + REQUIRE(sub_interp != main_interp); + + py::list(py::module_::import("sys").attr("path")).append(py::str(".")); + py::module_::import("datetime"); + py::module_::import("threading"); + py::module_::import("external_module"); + + { + py::subinterpreter_scoped_activate main(py::subinterpreter::main()); + REQUIRE(PyInterpreterState_Get() == main_interp); + + { + py::gil_scoped_release nogil{}; + { + py::gil_scoped_acquire yesgil{}; + REQUIRE(PyInterpreterState_Get() == main_interp); + } + } + + REQUIRE(PyInterpreterState_Get() == main_interp); + } + + REQUIRE(PyInterpreterState_Get() == sub_interp); + + { + py::gil_scoped_release nogil{}; + { + py::gil_scoped_acquire yesgil{}; + REQUIRE(PyInterpreterState_Get() == sub_interp); + } + } + + REQUIRE(PyInterpreterState_Get() == sub_interp); + } + + REQUIRE(PyInterpreterState_Get() == main_interp); + + { + py::gil_scoped_release nogil{}; + { + py::gil_scoped_acquire yesgil{}; + REQUIRE(PyInterpreterState_Get() == main_interp); + } + } + + REQUIRE(PyInterpreterState_Get() == main_interp); + + bool thread_result; + + { + thread_result = false; + py::gil_scoped_release nogil{}; + std::thread([&]() { + { + py::subinterpreter_scoped_activate ssa{sub}; + } + { + py::gil_scoped_acquire gil{}; + thread_result = (PyInterpreterState_Get() == main_interp); + } + }).join(); + } + REQUIRE(thread_result); + + { + thread_result = false; + py::gil_scoped_release nogil{}; + std::thread([&]() { + py::gil_scoped_acquire gil{}; + thread_result = (PyInterpreterState_Get() == main_interp); + }).join(); + } + REQUIRE(thread_result); + } + + REQUIRE(PyInterpreterState_Get() == main_interp); + unsafe_reset_internals_for_single_interpreter(); +} + +TEST_CASE("Multiple Subinterpreters") { + // Make sure the module is in the main interpreter and save its pointer + auto *main_ext = py::module_::import("external_module").ptr(); + auto main_int + = py::module_::import("external_module").attr("internals_at")().cast(); + py::module_::import("external_module").attr("multi_interp") = "1"; + + { + py::subinterpreter si1 = py::subinterpreter::create(); + std::unique_ptr psi2; + + PyObject *sub1_ext = nullptr; + PyObject *sub2_ext = nullptr; + uintptr_t sub1_int = 0; + uintptr_t sub2_int = 0; + + { + py::subinterpreter_scoped_activate scoped(si1); + py::list(py::module_::import("sys").attr("path")).append(py::str(".")); + + // The subinterpreter has its own copy of this module which is completely separate from + // main + sub1_ext = py::module_::import("external_module").ptr(); + REQUIRE(sub1_ext != main_ext); + REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp")); + py::module_::import("external_module").attr("multi_interp") = "2"; + // The subinterpreter also has its own internals + sub1_int + = py::module_::import("external_module").attr("internals_at")().cast(); + REQUIRE(sub1_int != main_int); + + // while the old one is active, create a new one + psi2.reset(new py::subinterpreter(py::subinterpreter::create())); + } + + { + py::subinterpreter_scoped_activate scoped(*psi2); + py::list(py::module_::import("sys").attr("path")).append(py::str(".")); + + // The second subinterpreter is separate from both main and the other subinterpreter + sub2_ext = py::module_::import("external_module").ptr(); + REQUIRE(sub2_ext != main_ext); + REQUIRE(sub2_ext != sub1_ext); + REQUIRE_FALSE(py::hasattr(py::module_::import("external_module"), "multi_interp")); + py::module_::import("external_module").attr("multi_interp") = "3"; + // The subinterpreter also has its own internals + sub2_int + = py::module_::import("external_module").attr("internals_at")().cast(); + REQUIRE(sub2_int != main_int); + REQUIRE(sub2_int != sub1_int); + } + + { + py::subinterpreter_scoped_activate scoped(si1); + REQUIRE( + py::cast(py::module_::import("external_module").attr("multi_interp")) + == "2"); + } + + // out here we should be in the main interpreter, with the GIL, with the other 2 still + // alive + + auto post_int + = py::module_::import("external_module").attr("internals_at")().cast(); + // Make sure internals went back the way it was before + REQUIRE(main_int == post_int); + + REQUIRE(py::cast(py::module_::import("external_module").attr("multi_interp")) + == "1"); + } + + // now back to just main + + auto post_int + = py::module_::import("external_module").attr("internals_at")().cast(); + // Make sure internals went back the way it was before + REQUIRE(main_int == post_int); + + REQUIRE(py::cast(py::module_::import("external_module").attr("multi_interp")) + == "1"); + + unsafe_reset_internals_for_single_interpreter(); +} + +# ifdef Py_MOD_PER_INTERPRETER_GIL_SUPPORTED +TEST_CASE("Per-Subinterpreter GIL") { + auto main_int + = py::module_::import("external_module").attr("internals_at")().cast(); + + std::atomic started, sync, failure; + started = 0; + sync = 0; + failure = 0; + +// REQUIRE throws on failure, so we can't use it within the thread +# define T_REQUIRE(status) \ + do { \ + assert(status); \ + if (!(status)) \ + ++failure; \ + } while (0) + + auto &&thread_main = [&](int num) { + while (started == 0) + std::this_thread::sleep_for(std::chrono::microseconds(1)); + ++started; + + py::gil_scoped_acquire gil; + + // we have the GIL, we can access the main interpreter + auto t_int + = py::module_::import("external_module").attr("internals_at")().cast(); + T_REQUIRE(t_int == main_int); + py::module_::import("external_module").attr("multi_interp") = "1"; + + auto sub = py::subinterpreter::create(); + + { + py::subinterpreter_scoped_activate sguard{sub}; + + py::list(py::module_::import("sys").attr("path")).append(py::str(".")); + + // we have switched to the new interpreter and released the main gil + + // trampoline_module did not provide the per_interpreter_gil tag, so it cannot be + // imported + bool caught = false; + try { + py::module_::import("trampoline_module"); + } catch (pybind11::error_already_set &pe) { + T_REQUIRE(pe.matches(PyExc_ImportError)); + std::string msg(pe.what()); + T_REQUIRE(msg.find("does not support loading in subinterpreters") + != std::string::npos); + caught = true; + } + T_REQUIRE(caught); + + // widget_module did provide the per_interpreter_gil tag, so it this does not throw + try { + py::module_::import("widget_module"); + caught = false; + } catch (pybind11::error_already_set &) { + caught = true; + } + T_REQUIRE(!caught); + + // widget_module did provide the per_interpreter_gil tag, so it this does not throw + py::module_::import("widget_module"); + + T_REQUIRE(!py::hasattr(py::module_::import("external_module"), "multi_interp")); + py::module_::import("external_module").attr("multi_interp") = std::to_string(num); + + // wait for something to set sync to our thread number + // we are holding our subinterpreter's GIL + while (sync != num) + std::this_thread::sleep_for(std::chrono::microseconds(1)); + + // now change it so the next thread can move on + ++sync; + + // but keep holding the GIL until after the next thread moves on as well + while (sync == num + 1) + std::this_thread::sleep_for(std::chrono::microseconds(1)); + + // one last check before quitting the thread, the internals should be different + auto sub_int + = py::module_::import("external_module").attr("internals_at")().cast(); + T_REQUIRE(sub_int != main_int); + } + }; +# undef T_REQUIRE + + std::thread t1(thread_main, 1); + std::thread t2(thread_main, 2); + + // we spawned two threads, at this point they are both waiting for started to increase + ++started; + + // ok now wait for the threads to start + while (started != 3) + std::this_thread::sleep_for(std::chrono::microseconds(1)); + + // we still hold the main GIL, at this point both threads are waiting on the main GIL + // IN THE CASE of free threading, the threads are waiting on sync (because there is no GIL) + + // IF the below code hangs in one of the wait loops, then the child thread GIL behavior did not + // function as expected. + { + // release the GIL and allow the threads to run + py::gil_scoped_release nogil; + + // the threads are now waiting on the sync + REQUIRE(sync == 0); + + // this will trigger thread 1 and then advance and trigger 2 and then advance + sync = 1; + + // wait for thread 2 to advance + while (sync != 3) + std::this_thread::sleep_for(std::chrono::microseconds(1)); + + // we know now that thread 1 has run and may be finishing + // and thread 2 is waiting for permission to advance + + // so we move sync so that thread 2 can finish executing + ++sync; + + // now wait for both threads to complete + t1.join(); + t2.join(); + } + + // now we have the gil again, sanity check + REQUIRE(py::cast(py::module_::import("external_module").attr("multi_interp")) + == "1"); + + unsafe_reset_internals_for_single_interpreter(); + + // make sure nothing unexpected happened inside the threads, now that they are completed + REQUIRE(failure == 0); +} +# endif // Py_MOD_PER_INTERPRETER_GIL_SUPPORTED + +#endif // PYBIND11_SUBINTERPRETER_SUPPORT