Clarification on what is needed for advancement toward per-interpreter GIL.

[ericsnowcurrently]

(FYI, I've spoken with @brettcannon about all this previously.)

I've been looking over what needs to be done to get to per-interpreter GIL, as well as concerns folks have brought up about the project. There's a chance things could move along fast enough to reach per-interpreter GIL in time for 3.11b1, so I want to see what the steering council thinks needs approval first (i.e. a PEP).

Some things to note:

• the discrete, mostly parallel, steps to get there are pretty well-understood
• none are very big
• there are just a lot of them
• nearly all are worth doing regardless of per-interpreter GIL
• few are controversial on their own that I'm aware of
• none will break backward compatibility
• none will touch the public C-API (outside what is covered in existing PEPs)

So it isn't so clear to me what value another PEP provides. Aside from existing PEPs, I could imagine possibly the following, but I'm not sure any of them provide much value.

• immortal objects
• making the GIL per-interpreter (we'll likely do nearly everything else necessary for that, so the actual change involved would be a few lines of code. Would a decision be necessary for that?)
• (informational) about the impact of per-interpreter GIL (the change in behavior and the impact on extension modules)

FYI, I started python/peps#2212 as a sort of dump of the whole thing, but it covered too much. Would it make more sense to add a page to the devguide about the project instead of a PEP?

Anyway, I figured I'd get your feedback sooner rather than later. 🙂

[ericsnowcurrently]

Here are the specific things that would need to be done before beta1:

• convert remaining stdlib extension modules to PEP 630
  • see discussion about SC concerns in Request for SC Comment: How to proceed with applying PEP 630 (Isolating Modules) in the CPython standard library #99
• move string objects generated by argument clinic to _PyRuntimeState
• move other PyObject * static variables to _PyRuntimeState (except for static types)
  • this is something like 100
• move other mutable static variables to _PyRuntimeState
  • this is less than 100
• solve the challenge of objects exposed in C-API (incl. limited API)
  • this will be either "immutable objects" or an alternative for which I have a proof-of-concept
• move as many objects as necessary from _PyRuntimeState to PyInterpreterState, along with most of the other fields in _PyRuntimeState
• move the GIL to PyInterpreterState
• enforce restrictions on how extensions are imported in subinterpreters
• have a good plan for helping large extensions (e.g. numpy) to support subinterpreters

The only (mild) controversy I'm aware of is with the project as a whole, including cases where some folks were premature in moving some objects to per-interpreter state.

---

I've written down a more concrete, exhaustive (though probably still slightly incomplete) TODO list here:

ericsnowcurrently/multi-core-python#78

[gvanrossum]

Could we decouple this decision from PEP 554? IIUC that PEP is about the specific API to offer for communicating between subinterpreters (which is important when we have separate GILs since you can't share objects). It looks rather old and I'm not sure I like the specifics of the proposed API that much. I would rather not have it accepted (or have it very provisionally accepted) than have to live with it since we were in a rush to get all this done before 3.11b1.

[erlend-aasland]

The only (mild) controversy I'm aware of is with the project as a whole, including cases where some folks were premature in moving some objects to per-interpreter state.

There is also some controversy (of varying degrees) when it comes to applying PEP 630 to the stdlib. See also #99

[ericsnowcurrently]

Could we decouple this decision from PEP 554?

+1

IIUC that PEP is about the specific API to offer for communicating between subinterpreters (which is important when we have separate GILs since you can't share objects). It looks rather old and I'm not sure I like the specifics of the proposed API that much. I would rather not have it accepted (or have it very provisionally accepted) than have to live with it since we were in a rush to get all this done before 3.11b1.

The PEP needs a slight update and then we can revisit its merits elsewhere.

[brettcannon]

I've added this to our agenda

[encukou]

I'd like to know what is the plan with static types (esp. in extensions). Can we keep them?
There's an issue to Convert static types to heap types. Is it part of this effort? Should it be finished, closed, or reverted?

(I should probably say in PEP 630 that it's not necessary to convert all static types to heap ones: many people think “apply PEP 630” implies that, and I would like to strongly suggest it for third-party libraries, but it makes less sense in stdlib.)

[gvanrossum]

(I should probably in PEP 630 that it's not necessary to convert all static types to heap ones: many people think “apply PEP 630” implies that, and I would like to strongly suggest it for third-party libraries, but it makes less sense in stdlib.)

Oh, I had missed that, and I think most people have missed that. There seems to be a systematic programme to do convert all static types somewhere in bpo.

[encukou]

There seems to be a systematic programme to do convert all static types somewhere in bpo.

That's the issue I linked. It's missing motivation/rationale.

[ericsnowcurrently]

I'd like to know what is the plan with static types (esp. in extensions). Can we keep them?

It mostly depends on if we end up with immortal objects or not. If we do then the story is relatively straight-forward. We'd have to update PyType_Type to defer some attributes (e.g. __subclasses__) to a per-interpreter storage (e.g. on the type object or on PyInterpreterState). Otherwise static types could mostly stay as-is.

Without immortal objects we would have to have a distinct object per interpreter for each static type. I expect in the core (including builtin modules) we would leave the static types alone and use them only for the main interpreter. Then for subinterpreters we would make a copy (and do some fixups).

So I suppose "is a PEP needed for immortal objects?" is the most pressing question. (Of course, I'd still like feedback on the other aspects of per-interpreter GIL. 🙂)

There's an issue to Convert static types to heap types. Is it part of this effort?

It is sort of part of this effort. 🙂 Regardless of how, all the global variables in our extension modules needs to be made per-interpreter (or at least safe to share between interpreters without a GIL). The work you are asking about helps and is probably the best solution. It just isn't necessary for per-interpreter GIL.

Also, IIUC the work on extension modules started many years ago with PEP 384, so it isn't new. However, the effort accelerated in the last couple years under @vstinner's push, at least in part motivated by the goal of a per-interpreter GIL.

Should it be finished, closed, or reverted?

I expect that depends on the outcome of #99.

[ericsnowcurrently]

It mostly depends on if we end up with immortal objects or not.

FYI, yesterday I ran the benchmarks on Eddie's branch and found a 4% slowdown. There are a number of strategies that will help us recover that, hopefully back to performance-neutral.

[encukou]

Also, IIUC the work on extension modules started many years ago with PEP 384, so it isn't new. However, the effort accelerated in the last couple years under @vstinner's push, at least in part motivated by the goal of a per-interpreter GIL.

The problem I see is that “the work” could include changes done without a good reason.
just because it .
Converting to heap types was part of an earlier plan and it seemed to benefit other similar work, so it was done en masse. But caused issues (with mutability and pickle-ability) that required workarounds which I don't think were ideal (new type flags). The conversion was done so fast that by release time it wasn't feasible to revert it.
And if immortal types are feasible, most of the conversion to heap types will have been pointless.

I don't want to blame anyone, and I know I am also part of the problem. But I'd like to untangle it and help avoid making similar mistakes in the future. (Hopefully without an overbooked SC as gatekeepers of innovation, too, but that's not where we are now.)

"is a PEP needed for immortal objects?" is the most pressing question

For me, yes. I would definitely like to see a write-up on the pros and cons and the whys, rather than having the info spread across comment threads. But I understand it's hard to do with an active research topic.

Hm, maybe “which parts of a PEP are needed?” could be a better question to ask than “is a PEP needed”? I often look at the PEP template to remind myself what to think about when considering a change, without actually writing a PEP. e.g. here the motivation is common to all steps, but it might be good to write down individual rationales, so we can check the “why”.
And maybe we also need a contingency plan section in PEPs, for changes that are too big to simply revert.

[erlend-aasland]

There's an issue to Convert static types to heap types. Is it part of this effort?

It is sort of part of this effort. 🙂 Regardless of how, all the global variables in our extension modules needs to be made per-interpreter (or at least safe to share between interpreters without a GIL). The work you are asking about helps and is probably the best solution. It just isn't necessary for per-interpreter GIL.

FTR, I took a quick glance at the extension modules with static types to see if they (they types) access global state:

Module	Types accessing global state?
Modules/_asynciomodule.c	yes
Modules/_collectionsmodule.c	no
Modules/_ctypes/_ctypes.c	yes
Modules/_datetimemodule.c	yes
Modules/_decimal/_decimal.c	yes
Modules/_elementtree.c	no
Modules/_pickle.c	yes
Modules/_zoneinfo.c	yes
Modules/itertoolsmodule.c	no
Modules/ossaudiodev.c	no
Modules/socketmodule.c	no

[ericsnowcurrently]

After a chat with @vstinner, I'm going to just go ahead with a PEP for immortal objects and one for per-interpreter GIL. Both should be fairly focused (unlike my old peps repo PR). We can continue any discussion on python-dev.