Schedule for a beta release of greenlet 2.0?

[vstinner]

Hi,

The stable branch of greenlet (greenlet 1.x) is currently not compatible with Python 3.11. The compatibility was added in the master branch, in the released greenlet 2.0.0 alpha 2. Problem: it's an alpha version, I would prefer to not use it in production.

In Fedora, we are working on replacing Python 3.10 with Python 3.11, and we would like to use greenlet 2.0. It would make me more comfortable if this branch would reach at least the beta phase. Is there a schedule for that? Can I help you?

It seems like gevent and eventlet just works with Python 3.11 beta1 and greenlet 2.0.0a2. There are compiler warnings #301 but Python 3.11 got fixed and the fix will be part of Python 3.11 beta2 scheduled next week.

Links about past issues.

• Nov 2021

  • pyperformance issue with greenlet: pyperformance doesn't work on the current Python 3.11 dev version: fail to install greenlet python/pyperformance#113

• December 2021

  • (1) First greenlet issue about Python 3.11: C extensions can't swap out live frames in CPython 3.11 #281
  • First greenlet fix for Python 3.11: 8d59c51 -- PR: Restore compatibility with Python 3.11 #280

• January 2022

  • (2) Second greenlet issue with Python 3.11: Compatibility with Python3.11 (again) #288

• March 2022

  • My greenlet fix for Python 3.11 alpha 6: 3cbbfd4 -- PR: Update for Python 3.11 alpha 6 #294
  • greenlet 2.0.0a2 release
  • pyperformance now uses greenlet 2.0.0a2: python/pyperformance@a591fb4

• April 2022

  • (3) Third greenlet issue with Python 3.11 (C++ compiler warning): py311 issues with const correctness #301 -- fixed in future Python 3.11 beta 2 (scheduled next week)

Both greenlet commtis are part of released greenlet 2.0.0a2.

[hroncok]

The stable branch of greenlet (greenlet 1.x) is currently not compatible with Python 3.11.

I have a patch for greenlet 1.x that appears to be working. Since there is no branch I could send PR to, I've added it here, in case other distributors find it useful. Unfortunately, it includes internal/pycore_frame.h.

diff --git a/src/greenlet/greenlet.c b/src/greenlet/greenlet.c
index f47bbf8..999989d 100644
--- a/src/greenlet/greenlet.c
+++ b/src/greenlet/greenlet.c
@@ -170,9 +170,11 @@ green_clear_exc(PyGreenlet* g)
 {
 #if GREENLET_PY37
     g->exc_info = NULL;
-    g->exc_state.exc_type = NULL;
     g->exc_state.exc_value = NULL;
+#if !GREENLET_PY311
+    g->exc_state.exc_type = NULL;
     g->exc_state.exc_traceback = NULL;
+#endif
     g->exc_state.previous_item = NULL;
 #else
     g->exc_type = NULL;
@@ -525,8 +527,12 @@ g_switchstack(void)
     { /* save state */
         PyGreenlet* current = ts_current;
         PyThreadState* tstate = PyThreadState_GET();
+#if GREENLET_PY311
+        current->recursion_depth = tstate->recursion_limit - tstate->recursion_remaining;
+#else
         current->recursion_depth = tstate->recursion_depth;
         current->top_frame = tstate->frame;
+#endif
 #if GREENLET_PY37
         current->context = tstate->context;
 #endif
@@ -551,6 +557,15 @@ g_switchstack(void)
          */
         current->cframe = tstate->cframe;
         ts__g_switchstack_use_tracing = tstate->cframe->use_tracing;
+#if GREENLET_PY311
+        current->current_frame = tstate->cframe->current_frame;
+        current->datastack_chunk = tstate->datastack_chunk;
+        current->datastack_top = tstate->datastack_top;
+        current->datastack_limit = tstate->datastack_limit;
+        PyFrameObject *frame = PyThreadState_GetFrame((PyThreadState *)tstate);
+        Py_XDECREF(frame); /* PyThreadState_GetFrame gives us a new reference. */
+        current->top_frame = frame;
+#endif
 #endif
     }
 
@@ -574,8 +589,6 @@ g_switchstack(void)
         PyGreenlet* target = ts_target;
         PyGreenlet* origin = ts_current;
         PyThreadState* tstate = PyThreadState_GET();
-        tstate->recursion_depth = target->recursion_depth;
-        tstate->frame = target->top_frame;
         target->top_frame = NULL;
 
 #if GREENLET_PY37
@@ -607,7 +620,16 @@ g_switchstack(void)
         */
         tstate->cframe->use_tracing = ts__g_switchstack_use_tracing;
 #endif
-
+#if GREENLET_PY311
+        tstate->recursion_remaining = tstate->recursion_limit - target->recursion_depth;
+        tstate->cframe->current_frame = target->current_frame;
+        tstate->datastack_chunk = target->datastack_chunk;
+        tstate->datastack_top = target->datastack_top;
+        tstate->datastack_limit = target->datastack_limit;
+#else
+        tstate->recursion_depth = target->recursion_depth;
+        tstate->frame = target->top_frame;
+#endif
         assert(ts_origin == NULL);
         Py_INCREF(target);
         ts_current = target;
@@ -810,7 +832,7 @@ static int GREENLET_NOINLINE(g_initialstub)(void* mark)
       We want to defer copying the state info until we're sure
       we need it and are in a stable place to do so.
     */
-    CFrame trace_info;
+    _PyCFrame trace_info;
 #endif
     /* save exception in case getattr clears it */
     PyErr_Fetch(&exc, &val, &tb);
@@ -875,7 +897,11 @@ static int GREENLET_NOINLINE(g_initialstub)(void* mark)
     }
     self->top_frame = NULL;
     green_clear_exc(self);
+#if GREENLET_PY311
+    self->recursion_depth = PyThreadState_GET()->recursion_limit - PyThreadState_GET()->recursion_remaining;
+#else
     self->recursion_depth = PyThreadState_GET()->recursion_depth;
+#endif
 
     /* restore arguments in case they are clobbered */
     ts_target = self;
@@ -1006,13 +1032,13 @@ green_new(PyTypeObject* type, PyObject* args, PyObject* kwds)
           it uses the ``root_cframe`` just to have something to put there.
           However, once the greenlet is actually switched to for the first
           time, ``g_initialstub`` (which doesn't actually "return" while the
-          greenlet is running) stores a new CFrame on its local stack, and
+          greenlet is running) stores a new _PyCFrame on its local stack, and
           copies the appropriate values from the currently running CFrame;
-          this is then made the CFrame for the newly-minted greenlet.
+          this is then made the _PyCFrame for the newly-minted greenlet.
           ``g_initialstub`` then proceeds to call ``glet.run()``, which
-          results in ``PyEval_...`` adding the CFrame to the list. Switches
+          results in ``PyEval_...`` adding the _PyCFrame to the list. Switches
           continue as normal. Finally, when the greenlet finishes, the call to
-          ``glet.run()`` returns and the CFrame is taken out of the linked
+          ``glet.run()`` returns and the _PyCFrame is taken out of the linked
           list and the stack value is now unused and free to expire.
         */
         ((PyGreenlet*)o)->cframe = &PyThreadState_GET()->root_cframe;
@@ -1121,9 +1147,11 @@ green_traverse(PyGreenlet* self, visitproc visit, void* arg)
     Py_VISIT(self->context);
 #endif
 #if GREENLET_PY37
-    Py_VISIT(self->exc_state.exc_type);
     Py_VISIT(self->exc_state.exc_value);
+#if !GREENLET_PY311
+    Py_VISIT(self->exc_state.exc_type);
     Py_VISIT(self->exc_state.exc_traceback);
+#endif
 #else
     Py_VISIT(self->exc_type);
     Py_VISIT(self->exc_value);
@@ -1159,9 +1187,11 @@ green_clear(PyGreenlet* self)
     Py_CLEAR(self->context);
 #endif
 #if GREENLET_PY37
-    Py_CLEAR(self->exc_state.exc_type);
     Py_CLEAR(self->exc_state.exc_value);
+#if !GREENLET_PY311
+    Py_CLEAR(self->exc_state.exc_type);
     Py_CLEAR(self->exc_state.exc_traceback);
+#endif
 #else
     Py_CLEAR(self->exc_type);
     Py_CLEAR(self->exc_value);
@@ -1253,9 +1283,11 @@ green_dealloc(PyGreenlet* self)
     Py_CLEAR(self->context);
 #endif
 #if GREENLET_PY37
-    Py_CLEAR(self->exc_state.exc_type);
     Py_CLEAR(self->exc_state.exc_value);
+#if !GREENLET_PY311
+    Py_CLEAR(self->exc_state.exc_type);
     Py_CLEAR(self->exc_state.exc_traceback);
+#endif
 #else
     Py_CLEAR(self->exc_type);
     Py_CLEAR(self->exc_value);
diff --git a/src/greenlet/greenlet.h b/src/greenlet/greenlet.h
index 830bef8..4a26ace 100644
--- a/src/greenlet/greenlet.h
+++ b/src/greenlet/greenlet.h
@@ -14,6 +14,15 @@ extern "C" {
 /* This is deprecated and undocumented. It does not change. */
 #define GREENLET_VERSION "1.0.0"
 
+#if PY_VERSION_HEX >= 0x30B00A6
+# define GREENLET_PY311 1
+/* We need this for _PyInterpreterFrame apparently */
+#include <internal/pycore_frame.h>
+#else
+#define GREENLET_PY311 0
+#define _PyCFrame CFrame
+#endif
+
 typedef struct _greenlet {
     PyObject_HEAD
     char* stack_start;
@@ -25,6 +34,12 @@ typedef struct _greenlet {
     PyObject* run_info;
     struct _frame* top_frame;
     int recursion_depth;
+#if GREENLET_PY311
+    _PyInterpreterFrame *current_frame;
+    _PyStackChunk *datastack_chunk;
+    PyObject **datastack_top;
+    PyObject **datastack_limit;
+#endif
     PyObject* weakreflist;
 #if PY_VERSION_HEX >= 0x030700A3
     _PyErr_StackItem* exc_info;
@@ -39,7 +54,7 @@ typedef struct _greenlet {
     PyObject* context;
 #endif
 #if PY_VERSION_HEX >= 0x30A00B1
-    CFrame* cframe;
+    _PyCFrame* cframe;
 #endif
 } PyGreenlet;
 

[vstinner]

I have a patch for greenlet 1.x that appears to be working.

For that, I created:

• issue Consider releasing a new greenlet 1.1.3 version compatible with Python 3.11 #305 "Consider releasing a new greenlet 1.1.3 version compatible with Python 3.11"
• PR Closes #305: Add Python 3.11 support #306: "Add Python 3.11 support" (to greenlet 1.1.2)

[jamadden]

Thanks to everyone for their help. Version 1.1.3 and version 2.0.0 with Python 3.11 support have been released. (gevent 21.10.2 is required for greenlet 2.0.)