From fea85c5613a60d26b2afb7b81feb64f7ed172f95 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 26 Jul 2023 14:14:34 -0600 Subject: [PATCH 01/13] Add an entry for PyInterpreterConfig. --- Doc/c-api/init.rst | 81 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 48cea69c96940f..8706cd3f32b630 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1223,6 +1223,87 @@ You can switch between sub-interpreters using the :c:func:`PyThreadState_Swap` function. You can create and destroy them using the following functions: +.. c:type:: PyInterpreterConfig + + Structure containing most parameters to configure a sub-interpreter. + Its values are used only in :c:func:`Py_NewInterpreterFromConfig` and + never modified by the runtime. + + .. versionadded:: 3.12 + + Structure fields: + + .. c:member:: int use_main_obmalloc + + If this is ``0`` then the sub-interpreter will use its own + "object" allocator state. + Otherwise it will use (share) the main interpreter's. + + If this is ``0`` then + :c:member:`PyInterpreterConfig.check_multi_interp_extensions` + must be ``1`` (non-zero). + If this is ``1`` then :c:member:`PyInterpreterConfig.gil` + must not be ``PyInterpreterConfig_OWN_GIL``. + + .. c:member:: int allow_fork + + If this is ``0`` then the runtime will not support forking the + process in any thread where the sub-interpreter is currently active. + Otherwise fork is unrestricted. + + Note that the :mod:`subprocess` module still works + when fork is disallowed. + + .. c:member:: int allow_exec + + If this is ``0`` then the runtime will not support replacing the + current process via exec (e.g. :func:`os.execv`) in any thread + where the sub-interpreter is currently active. + Otherwise exec is unrestricted. + + Note that the :mod:`subprocess` module still works + when exec is disallowed. + + .. c:member:: int allow_threads + + If this is ``0`` then the sub-interpreter's :mod:`threading` module + won't create threads. + Otherwise threads are allowed. + + .. c:member:: int allow_daemon_threads + + If this is ``0`` then the sub-interpreter's :mod:`threading` module + won't create daemon threads. + Otherwise daemon threads are allowed (as long as + :c:member:`PyInterpreterConfig.allow_threads` is non-zero). + + .. c:member:: int check_multi_interp_extensions + + If this is ``0`` then all extension modules may be imported, + including legacy (single-phase init) modules, + in any thread where the sub-interpreter is currently active. + Otherwise only multi-phase init extension modules + (see :ref:`Isolating Extension Modules`) may be imported. + + This must be ``1`` (non-zero) if + :c:member:`PyInterpreterConfig.use_main_obmalloc` is ``0``. + + .. c:member:: int gil + + This determines the operation of the GIL for the sub-interpreter. + It may be one of the following: + + - ``PyInterpreterConfig_DEFAULT_GIL``: use the default selection + (``PyInterpreterConfig_SHARED_GIL``) + - ``PyInterpreterConfig_SHARED_GIL``: use (share) the main + interpreter's GIL + - ``PyInterpreterConfig_OWN_GIL``: use the sub-interpreter's + own GIL + + If this is ``PyInterpreterConfig_OWN_GIL`` then + :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``. + + .. c:function:: PyThreadState* Py_NewInterpreter() .. index:: From c635e18dec97b0fc38c089a5297ab541302e0f16 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 26 Jul 2023 14:26:20 -0600 Subject: [PATCH 02/13] Add an entry for Py_NewInterpreterFromConfig(). --- Doc/c-api/init.rst | 45 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 8706cd3f32b630..c93d7340eb1c9f 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1304,7 +1304,7 @@ function. You can create and destroy them using the following functions: :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``. -.. c:function:: PyThreadState* Py_NewInterpreter() +.. c:function:: PyThreadState* Py_NewInterpreterFromConfig(PyInterpreterConfig *config) .. index:: pair: module; builtins @@ -1314,6 +1314,10 @@ function. You can create and destroy them using the following functions: single: stderr (in module sys) single: stdin (in module sys) + Create a new sub-interpreter using the given config. This is like + :c:func:`Py_NewInterpreter` except you control the options with which + the interpreter is initialized. + Create a new sub-interpreter. This is an (almost) totally separate environment for the execution of Python code. In particular, the new interpreter has separate, independent versions of all imported modules, including the @@ -1335,6 +1339,27 @@ function. You can create and destroy them using the following functions: other Python/C API functions, there needn't be a current thread state on entry.) + .. versionadded:: 3.12 + + Sub-interpreters are most effective when isolated from each other, + with certain functionality restricted:: + + PyInterpreterConfig config = { + .use_main_obmalloc = 0, + .allow_fork = 0, + .allow_exec = 0, + .allow_threads = 1, + .allow_daemon_threads = 0, + .check_multi_interp_extensions = 1, + .gil = PyInterpreterConfig_OWN_GIL, + }; + PyThreadState *tstate = Py_NewInterpreterFromConfig(&config); + + Note that the config is used only briefly and does not get modified. + During initialization the config's values are converted into various + :c:type:`PyInterpreterState` values. A read-only copy of the config + may be stored internally on the :c:type:`PyInterpreterState`. + .. index:: single: Py_FinalizeEx() single: Py_Initialize() @@ -1368,6 +1393,24 @@ function. You can create and destroy them using the following functions: .. index:: single: close() (in module os) +.. c:function:: PyThreadState* Py_NewInterpreter() + + .. index:: + pair: module; builtins + pair: module; __main__ + pair: module; sys + single: stdout (in module sys) + single: stderr (in module sys) + single: stdin (in module sys) + + Create a new sub-interpreter. This is essentially just a wrapper + around :c:func:`Py_NewInterpreterFromConfig` with a config that + preserves the existing behavior. The result is an unisolated + sub-interpreter that shares the main interpreter's GIL, allows + fork/exec, allows daemon threads, and allows single-phase init + modules. + + .. c:function:: void Py_EndInterpreter(PyThreadState *tstate) .. index:: single: Py_FinalizeEx() From 4abebc90673518aba00b4a0d9e2a3f7cc8e7e21a Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 26 Jul 2023 15:36:43 -0600 Subject: [PATCH 03/13] Add a section about the consequences of per-interpreter GIL. --- Doc/c-api/init.rst | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index c93d7340eb1c9f..fa2b2decabcd8f 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1424,6 +1424,45 @@ function. You can create and destroy them using the following functions: haven't been explicitly destroyed at that point. +A Per-Interpreter GIL +--------------------- + +Using :c:func:`Py_NewInterpreterFromConfig` you can create +a sub-interpreter that is completely isolated from other interpreters, +including having its own GIL. The most important benefit of this +isolation is that such an interpreter can execute Python code without +being blocked by other interpreters or blocking any others. Thus a +single Python process can truly take advantage of multiple CPU cores +when running Python code. The isolation also encourages a different +approach to concurrency than that of just using threads. +(See :pep:`554`.) + +Using an isolated interpreter requires vigilance in preserving that +isolation. That especially means not sharing any objects or mutable +state without guarantees about thread-safety. Even objects that are +otherwise immutable (e.g. ``None``, ``(1, 5)``) can't normally be shared +because of the refcount. One simple but less-efficient approach around +this is to use a global lock around all use of some state (or object). +Alternately, effectively immutable objects (like integers or strings) +can be made safe in spite of their refcounts by making them "immortal". +In fact, this has been done for the builtin singletons, small integers, +and a number of other builtin objects. + +If you preserve isolation then you will have access to proper multi-core +computing without the complications that come with free-threading. +Failure to preserve isolation will expose you to the full consequences +of free-threading, including races and hard-to-debug crashes. + +Aside from that, one of the main challenges of using multiple isolated +interpreters is how to communicate between them safely (not break +isolation) and efficiently. The runtime and stdlib do not provide +any standard approach to this yet. A future stdlib module would help +mitigate the effort of preserving isolation and expose effective tools +for communicating (and sharing) data between interpreters. + +.. versionadded:: 3.12 + + Bugs and caveats ---------------- From a160739e1a69d077615efc87169cf74c12ad7821 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 26 Jul 2023 16:29:59 -0600 Subject: [PATCH 04/13] Clarify about the GIL/current tstate for Py_NewInterpreter() and Py_EndInterpreter(). --- Doc/c-api/init.rst | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index fa2b2decabcd8f..cc3e9bc5cf550b 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1333,11 +1333,16 @@ function. You can create and destroy them using the following functions: Note that no actual thread is created; see the discussion of thread states below. If creation of the new interpreter is unsuccessful, ``NULL`` is returned; no exception is set since the exception state is stored in the - current thread state and there may not be a current thread state. (Like all - other Python/C API functions, the global interpreter lock must be held before - calling this function and is still held when it returns; however, unlike most - other Python/C API functions, there needn't be a current thread state on - entry.) + current thread state and there may not be a current thread state. + + Like all other Python/C API functions, the global interpreter lock + must be held before calling this function and is still held when it + returns. Likewise a current thread state must be set on entry. On + success, the returned thread state will be set as current. If the + sub-interpreter is created with its own GIL then the GIL of the + calling interpreter will be released. When the function returns, + the new interpreter's GIL will be held by the current thread and + the previously interpreter's GIL will remain released here. .. versionadded:: 3.12 @@ -1415,12 +1420,15 @@ function. You can create and destroy them using the following functions: .. index:: single: Py_FinalizeEx() - Destroy the (sub-)interpreter represented by the given thread state. The given - thread state must be the current thread state. See the discussion of thread - states below. When the call returns, the current thread state is ``NULL``. All - thread states associated with this interpreter are destroyed. (The global - interpreter lock must be held before calling this function and is still held - when it returns.) :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that + Destroy the (sub-)interpreter represented by the given thread state. + The given thread state must be the current thread state. See the + discussion of thread states below. When the call returns, + the current thread state is ``NULL``. All thread states associated + with this interpreter are destroyed. The global interpreter lock + used by the target interpreter must be held before calling this + function. No GIL is held when it returns. + + :c:func:`Py_FinalizeEx` will destroy all sub-interpreters that haven't been explicitly destroyed at that point. From badd3d6239cddc5747d2d0103b645e9fe25f62b9 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Wed, 26 Jul 2023 16:33:14 -0600 Subject: [PATCH 05/13] Add a NEWS entry. --- .../2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst diff --git a/Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst b/Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst new file mode 100644 index 00000000000000..038f9e68a5422a --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2023-07-26-16-33-04.gh-issue-107305.qB2LS4.rst @@ -0,0 +1,3 @@ +Add documentation for :c:type:`PyInterpreterConfig` and +:c:func:`Py_NewInterpreterFromConfig`. Also clarify some of the nearby docs +relative to per-interpreter GIL. From 10d40b6ee5eec250f7fe95a5d0f2e26f1d3c4219 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 12:12:33 -0600 Subject: [PATCH 06/13] Use the c:macro role. --- Doc/c-api/init.rst | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 904e095540d6ca..2f23ec469b7633 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1246,7 +1246,7 @@ function. You can create and destroy them using the following functions: :c:member:`PyInterpreterConfig.check_multi_interp_extensions` must be ``1`` (non-zero). If this is ``1`` then :c:member:`PyInterpreterConfig.gil` - must not be ``PyInterpreterConfig_OWN_GIL``. + must not be :c:macro:`PyInterpreterConfig_OWN_GIL`. .. c:member:: int allow_fork @@ -1296,14 +1296,21 @@ function. You can create and destroy them using the following functions: This determines the operation of the GIL for the sub-interpreter. It may be one of the following: - - ``PyInterpreterConfig_DEFAULT_GIL``: use the default selection - (``PyInterpreterConfig_SHARED_GIL``) - - ``PyInterpreterConfig_SHARED_GIL``: use (share) the main - interpreter's GIL - - ``PyInterpreterConfig_OWN_GIL``: use the sub-interpreter's - own GIL + .. c:namespace:: NULL - If this is ``PyInterpreterConfig_OWN_GIL`` then + .. c:macro:: PyInterpreterConfig_DEFAULT_GIL + + Use the default selection (:c:macro:`PyInterpreterConfig_SHARED_GIL`). + + .. c:macro:: PyInterpreterConfig_SHARED_GIL + + Use (share) the main interpreter's GIL. + + .. c:macro:: PyInterpreterConfig_OWN_GIL + + Use the sub-interpreter's own GIL. + + If this is :c:macro:`PyInterpreterConfig_OWN_GIL` then :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``. From 919382d53044131fb358f0c9a90428e397a2d1e8 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 12:17:34 -0600 Subject: [PATCH 07/13] Drop the struct name when rendered. --- Doc/c-api/init.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 2f23ec469b7633..1e04f06a0b3c33 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1243,7 +1243,7 @@ function. You can create and destroy them using the following functions: Otherwise it will use (share) the main interpreter's. If this is ``0`` then - :c:member:`PyInterpreterConfig.check_multi_interp_extensions` + :c:member:`~PyInterpreterConfig.check_multi_interp_extensions` must be ``1`` (non-zero). If this is ``1`` then :c:member:`PyInterpreterConfig.gil` must not be :c:macro:`PyInterpreterConfig_OWN_GIL`. @@ -1278,7 +1278,7 @@ function. You can create and destroy them using the following functions: If this is ``0`` then the sub-interpreter's :mod:`threading` module won't create daemon threads. Otherwise daemon threads are allowed (as long as - :c:member:`PyInterpreterConfig.allow_threads` is non-zero). + :c:member:`~PyInterpreterConfig.allow_threads` is non-zero). .. c:member:: int check_multi_interp_extensions @@ -1289,7 +1289,7 @@ function. You can create and destroy them using the following functions: (see :ref:`Isolating Extension Modules`) may be imported. This must be ``1`` (non-zero) if - :c:member:`PyInterpreterConfig.use_main_obmalloc` is ``0``. + :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``. .. c:member:: int gil From c9a216267c0a0d615101a8fb2964bba1b4d517d3 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 12:19:53 -0600 Subject: [PATCH 08/13] Add a missing "void". --- Doc/c-api/init.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 1e04f06a0b3c33..ad6451fe7063a5 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1408,7 +1408,7 @@ function. You can create and destroy them using the following functions: .. index:: single: close() (in module os) -.. c:function:: PyThreadState* Py_NewInterpreter() +.. c:function:: PyThreadState* Py_NewInterpreter(void) .. index:: pair: module; builtins From 964259bdef5039dc67cd5f3ecb9911ae7788012f Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 12:25:52 -0600 Subject: [PATCH 09/13] Drop some old text. --- Doc/c-api/init.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index ad6451fe7063a5..202e7f9d45f6ad 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1324,10 +1324,6 @@ function. You can create and destroy them using the following functions: single: stderr (in module sys) single: stdin (in module sys) - Create a new sub-interpreter using the given config. This is like - :c:func:`Py_NewInterpreter` except you control the options with which - the interpreter is initialized. - Create a new sub-interpreter. This is an (almost) totally separate environment for the execution of Python code. In particular, the new interpreter has separate, independent versions of all imported modules, including the @@ -1338,6 +1334,9 @@ function. You can create and destroy them using the following functions: ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying file descriptors). + The given config controls the options with which the interpreter + is initialized. + The return value points to the first thread state created in the new sub-interpreter. This thread state is made in the current thread state. Note that no actual thread is created; see the discussion of thread states From db5ecf5a2770afed982ed959a5af80d2d586ed6c Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 12:34:04 -0600 Subject: [PATCH 10/13] Fix the signature of Py_NewInterpreterFromConfig(). --- Doc/c-api/init.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 202e7f9d45f6ad..66ec5ec2667244 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1314,7 +1314,7 @@ function. You can create and destroy them using the following functions: :c:member:`PyInterpreterConfig.use_main_obmalloc` must be ``0``. -.. c:function:: PyThreadState* Py_NewInterpreterFromConfig(PyInterpreterConfig *config) +.. c:function:: PyStatus Py_NewInterpreterFromConfig(PyThreadState **tstate_p, const PyInterpreterConfig *config) .. index:: pair: module; builtins @@ -1337,11 +1337,13 @@ function. You can create and destroy them using the following functions: The given config controls the options with which the interpreter is initialized. - The return value points to the first thread state created in the new + Upon success, the ``tstate_p`` arg will be set to the first thread state + created in the new sub-interpreter. This thread state is made in the current thread state. Note that no actual thread is created; see the discussion of thread states - below. If creation of the new interpreter is unsuccessful, ``NULL`` is - returned; no exception is set since the exception state is stored in the + below. If creation of the new interpreter is unsuccessful, + ``tstate_p`` is set to ``NULL``; + no exception is set since the exception state is stored in the current thread state and there may not be a current thread state. Like all other Python/C API functions, the global interpreter lock From a5b19f5366dbed6686638dcc4ac72306304e02a0 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 12:58:06 -0600 Subject: [PATCH 11/13] Fix a ref. --- Doc/c-api/init.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 66ec5ec2667244..06243fff3fe78a 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1286,7 +1286,7 @@ function. You can create and destroy them using the following functions: including legacy (single-phase init) modules, in any thread where the sub-interpreter is currently active. Otherwise only multi-phase init extension modules - (see :ref:`Isolating Extension Modules`) may be imported. + (see :pep:`489`) may be imported. This must be ``1`` (non-zero) if :c:member:`~PyInterpreterConfig.use_main_obmalloc` is ``0``. From 4d81e60743f2345fb9d03bf3e0f08dd221d32ad2 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 13:22:55 -0600 Subject: [PATCH 12/13] Drop the struct name when rendered. --- Doc/c-api/init.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 06243fff3fe78a..0341dbf9380784 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1245,7 +1245,7 @@ function. You can create and destroy them using the following functions: If this is ``0`` then :c:member:`~PyInterpreterConfig.check_multi_interp_extensions` must be ``1`` (non-zero). - If this is ``1`` then :c:member:`PyInterpreterConfig.gil` + If this is ``1`` then :c:member:`~PyInterpreterConfig.gil` must not be :c:macro:`PyInterpreterConfig_OWN_GIL`. .. c:member:: int allow_fork From a38bb8987e5f39ec5e3d67f896c33da52d42fd94 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 28 Jul 2023 13:24:23 -0600 Subject: [PATCH 13/13] Fix formatting for function arguments. --- Doc/c-api/init.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 0341dbf9380784..60912a1ecaede2 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1334,15 +1334,15 @@ function. You can create and destroy them using the following functions: ``sys.stdout`` and ``sys.stderr`` (however these refer to the same underlying file descriptors). - The given config controls the options with which the interpreter + The given *config* controls the options with which the interpreter is initialized. - Upon success, the ``tstate_p`` arg will be set to the first thread state + Upon success, *tstate_p* will be set to the first thread state created in the new sub-interpreter. This thread state is made in the current thread state. Note that no actual thread is created; see the discussion of thread states below. If creation of the new interpreter is unsuccessful, - ``tstate_p`` is set to ``NULL``; + *tstate_p* is set to ``NULL``; no exception is set since the exception state is stored in the current thread state and there may not be a current thread state.