PEP for Per-Interpreter GIL

[ericsnowcurrently]

At this point it makes sense to have a PEP about the plan for a per-interpreter GIL. This will be a companion to PEP 554. The PEP will cover the following (but more focused than #77):

• discuss about interpreter isolation
  • what are we isolating?
  • benefits and costs
  • extent
    • cannot share mutable state
    • cannot share (mutable) objects; all objects are mutable due to refcount
    • summary of global variables that must be fixed (e.g. made per-interpreter)
  • explain how most of work is worth doing anyway
  • challenges
    • objects exposed in public C-API, incl. limited API and stable ABI
    • breakage due to existing runtime bugs that get exposed by subinterpreters
    • breakage when moving extensions to PEP 630 due to existing bugs (see python/steering-council#99)
  • summarize immortal objects (see the PEP and faster-cpython/ideas#276)
    • allows us to share otherwise immutable objects (e.g. str, None)
    • cover the alternatives (e.g. lookup funcs + C-API hacks)
    • for now, assume that the immortal objects PEP will be accepted (if it isn't, update this PEP with the alternatives)
• discuss impact
• mention similarities (and differences) with Javascript's worker threads
  • I'd rather avoid any confusion, but @vstinner even suggested changing the term from "subinterpreter" to "worker thread", in the context of the use case I have in mind
• outline the plan (see #78)
  1. prep for per-interpreter
     • fix all globals in core/builtins (e.g. consolidate to _PyRuntimeState, use atomics, replace)
     • fix all globals in stdlib extensions (e.g. use module state)
     • explicitly support subinterpreters in all stdlib extension modules, AKA PEP 630 (see bpo-40077 & the SC discussion)
     • deal with objects held by static types (e.g. __subclasses__)
  2. update import system to enforce restrictions on extension modules
  3. move most data from _PyRuntimeState to PyInterpreterState
  4. move the GIL to PyInterpreterState
  • make the PyConfig changes
• specified changes
  • C-API: PyConfig options:
    • deprecate the existing "isolated" option (too general)
    • add: disallow fork
    • add: disallow additional threads
    • add: always run in own thread
    • add: share the GIL with the main interpreter
    • others?
    • default for all the above: behave the same as <3.8 (AKA "legacy mode")
  • C-API: add anything missing that is necessary to implement PEP 554 as an extension on PyPI (in case it doesn't make it in time)
  • update import system to enforce restrictions on
  • others?

I'm not quite sure, but I'll probably post this PEP before we reach a conclusion on the immortal objects PEP.

[ericsnowcurrently]

I'll probably work on separate PEPs for the following:

• a public C-API similar to _Py_IDENTIFIER() for caching strings (e.g. in module state)
• a public C-API for looking up global objects (e.g. global strings) and deprecate exposed objects in C-API

Likewise for the following separate work in the effort to help extensions support subinterpreters:

• some sort of compatibility shim for static types (or a tool to convert them to module-state heap types)
• ...

[ericsnowcurrently]

Note that, as with all PEPs, this won't be finished quickly. See faster-cpython/ideas#276 (comment).

[ericsnowcurrently]

What happens if we remove the GIL? The nogil branch is taking shape very nicely.

Yeah, that's a neat project. There shouldn't be any significant conflicts in the runtime design. Conceptually the projects coexist just fine. The real objective with per-interpreter GIL is maximum isolation between interpreters, where the GIL is essentially the last piece of state to isolate. That will remain even if the GIL is removed.

Note, however, that one of the main benefits of interpreter isolation is that running code concurrently in different interpreters allows you to achieve true multi-core parallelism. That is also the main objective of the nogil project. The difference is in the concurrency model that each supports. For per-interpreter GIL it is something like CSP or the actor model. For nogil it is threading. Personally, I find the former to fit my brain much better than threading. (See PEP 554.) I suppose you could ask, "why would we need one if we have the other?" I'd say there is value to each. All that assumes that both projects are actually in a position to make it into CPython. 🙂

[ericsnowcurrently]

The real objective with per-interpreter GIL is maximum isolation between interpreters

Hmm, maybe the PEP should emphasize isolation rather than the GIL...

[jakirkham]

Idk how subinterpreters are spun up, but there are often use cases where using multiprocessing (in particular forking) causes issues (for example some libraries are not fork-safe). So having a different mode of parallelism that can be used more safely with them would be helpful in its own right. Sorry if this is totally off-topic, just thought it might be an interesting data point 🙂

[ericsnowcurrently]

python/peps#2387