3.13.0rc1 regression in PyEval_GetLocals(): SystemError: Objects/dictobject.c:3774: bad argument to internal function

[mgorny]

Crash report

What happened?

After upgrading to CPython 3.13.0rc1, the gpgme test suite started failing. The tests fail with errors resembling:

Traceback (most recent call last):
  File "/tmp/gpgme-1.23.2/lang/python/tests/./final.py", line 24, in <module>
    import support
  File "/tmp/gpgme-1.23.2/lang/python/tests/support.py", line 60, in <module>
    assert_gpg_version((2, 1, 12))
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/tmp/gpgme-1.23.2/lang/python/tests/support.py", line 35, in assert_gpg_version
    c.engine_info.version).group(0)
    ^^^^^^^^^^^^^
  File "/tmp/gpgme-1.23.2/lang/python/python3.13-gpg/lib.linux-x86_64-cpython-313/gpg/core.py", line 1352, in engine_info
    infos = [i for i in self.get_engine_info() if i.protocol == p]
                        ~~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/gpgme-1.23.2/lang/python/python3.13-gpg/lib.linux-x86_64-cpython-313/gpg/core.py", line 1366, in get_engine_info
    return gpgme.gpgme_ctx_get_engine_info(self.wrapped)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/tmp/gpgme-1.23.2/lang/python/python3.13-gpg/lib.linux-x86_64-cpython-313/gpg/gpgme.py", line 880, in gpgme_ctx_get_engine_info
    return _gpgme.gpgme_ctx_get_engine_info(ctx)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "<frozen importlib._bootstrap>", line 649, in parent
SystemError: Objects/dictobject.c:3774: bad argument to internal function
FAIL: final.py

I've been able to bisect it to 233ed46 (#121869). However, on this commit I'm getting also a bunch of python3.13: Objects/typeobject.c:5257: _PyType_LookupRef: Assertion '!PyErr_Occurred()' failed. — they are fixed in some subsequent commit. But the final.py failure occurs from that commit to tip of 3.13 (fe65a8b).

Now, gpgme's build system is horrible. To reproduce, I've done the following:

1. Built CPython with --with-assertions.
2. Created ln -s python python3.13 symlink in the build directory.
3. Installed the gpgme C library 1.23.2 using the system package manager.
4. Then:

wget https://www.gnupg.org/ftp/gcrypt/gpgme/gpgme-1.23.2.tar.bz2
tar -xf gpgme-1.23.2.tar.bz2
cd gpgme-1.23.2
export PATH=${CPYTHON_BUILD_DIRECTORY}:$PATH
./configure  # will probably need some -dev packages
cd lang/python
make PYTHONS=python3.13 check

Note that in order to rebuild, you need to rm -rf python3.13-gpg.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.0rc1 (main, Aug 2 2024, 18:56:30) [GCC 14.2.0]

Linked PRs

• gh-122728: Fix SystemError in PyEval_GetLocals() #122735
• [3.13] gh-122728: Fix SystemError in PyEval_GetLocals() (GH-122735) #122757

[thesamesam]

cc @gaogaotiantian @ncoghlan

[hroncok]

We see this in Fedora as well.

https://bugzilla.redhat.com/show_bug.cgi?id=2303107

gpgme built with Python 3.13.0b4 exhibits the same behavior when run on Python 3.13.0rc1.

[hroncok]

Built CPython with --with-assertions.

Should not be necessary. We build Python without and the same SystemError happens.

[vstinner]

The problem is that PyDict_Update() is called with NULL as the first argument. It's a regression in PyEval_GetLocals().

gdb traceback:

(gdb) where
#0  dict_merge (interp=0x983f60 <_PyRuntime+88192>, a=a@entry=0x0, b=b@entry=0x7ffff7b34b50, override=override@entry=1) at Objects/dictobject.c:3796
#1  0x00000000004ece12 in PyDict_Update (a=a@entry=0x0, b=b@entry=0x7ffff7b34b50) at Objects/dictobject.c:3887
#2  0x0000000000607cbe in PyEval_GetLocals () at Python/ceval.c:2492
#3  0x00007ffff77ce622 in _gpg_wrap_result (fragile=fragile@entry=0x7ffff7835940, classname=classname@entry=0x7ffff77d05ab "EngineInfo") at python3.13-gpg/helpers.c:318

(gdb) frame 3
#3  0x00007ffff77ce622 in _gpg_wrap_result (fragile=fragile@entry=0x7ffff7835940, classname=classname@entry=0x7ffff77d05ab "EngineInfo") at python3.13-gpg/helpers.c:318
318	      results = PyImport_ImportModuleLevel("results", PyEval_GetGlobals(),
319	                                           PyEval_GetLocals(), from_list, 1);

(gdb) frame 2
#2  0x0000000000607cbe in PyEval_GetLocals () at Python/ceval.c:2492
2492	        if (PyDict_Update(ret, locals) < 0) {

(gdb) l 2488
2483	        PyObject *ret = f->f_locals_cache;
2484	        if (ret == NULL) {
2485	            PyObject *ret = PyDict_New();
2486	            if (ret == NULL) {
2487	                Py_DECREF(locals);
2488	                return NULL;
2489	            }
2490	            f->f_locals_cache = ret;
2491	        }
2492	        if (PyDict_Update(ret, locals) < 0) {

(gdb) p ret
$1 = (PyObject *) 0x0
(gdb) p locals
$2 = (PyObject *) 0x7ffff7b34b50

[vstinner]

Fixed. Thanks @mgorny and @hroncok for the bug report.