Skip to content

Regression in Python 3.14.0a7. #133352

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
felixxm opened this issue May 3, 2025 · 11 comments
Open

Regression in Python 3.14.0a7. #133352

felixxm opened this issue May 3, 2025 · 11 comments
Labels
3.14 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@felixxm
Copy link
Contributor

felixxm commented May 3, 2025 •
edited
Loading

Crash report

What happened?

I noticed a regression in Python 3.14.0 alpha 7. One of tests from Django test suite crashes test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test:

$ ./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
Fatal Python error: Segmentation fault

Current thread 0x00007858cbcc2b80 [python] (most recent call first):
  File "/django/django/test/runner.py", line 185 in _confirm_picklable
  File "/django/django/test/runner.py", line 215 in check_picklable
  File "/django/django/test/runner.py", line 294 in addError
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 246 in _addClassOrModuleLevelException
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 236 in _createClassOrModuleLevelException
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 176 in _handleClassSetUp
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 114 in run
  File "/django/tests/test_runner/test_parallel.py", line 225 in test_handle_add_error_before_first_test
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 615 in _callTestMethod
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 669 in run
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/case.py", line 725 in __call__
  File "/django/django/test/testcases.py", line 357 in _setup_and_call
  File "/django/django/test/testcases.py", line 302 in __call__
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 122 in run
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/suite.py", line 84 in __call__
  File ".pyenv/versions/3.14-dev/lib/python3.14/unittest/runner.py", line 259 in run
  File "/django/django/test/runner.py", line 1020 in run_suite
  File "/django/django/test/runner.py", line 1093 in run_tests
  File "/django/tests/./runtests.py", line 427 in django_tests
  File "/django/tests/./runtests.py", line 788 in <module>

Current thread's C stack trace (most recent call first):
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _Py_DumpStack+0x45 [0x7858cb8e3145]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2f85d4 [0x7858cb8f85d4]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x42520 [0x7858cb242520]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at PyObject_GC_UnTrack+0x19 [0x7858cb887269]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2e08a3 [0x7858cb8e08a3]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x10cb15 [0x7858cb70cb15]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x10d235 [0x7858cb70d235]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x18e862 [0x7858cb78e862]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x8ec9 [0x7858cb699059]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfea03 [0x7858cb6fea03]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1e48 [0x7858cb691fd8]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfc8ea [0x7858cb6fc8ea]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfcb40 [0x7858cb6fcb40]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x19d144 [0x7858cb79d144]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyObject_MakeTpCall+0x86 [0x7858cb6fa506]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1c1e [0x7858cb691dae]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfea03 [0x7858cb6fea03]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1e48 [0x7858cb691fd8]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x254fba [0x7858cb854fba]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfc8ea [0x7858cb6fc8ea]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0xfcb40 [0x7858cb6fcb40]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x19d144 [0x7858cb79d144]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyObject_MakeTpCall+0x86 [0x7858cb6fa506]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at _PyEval_EvalFrameDefault+0x1c1e [0x7858cb691dae]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at PyEval_EvalCode+0x1dd [0x7858cb854ced]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c57ad [0x7858cb8c57ad]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c7996 [0x7858cb8c7996]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at +0x2c7f3f [0x7858cb8c7f3f]
  Binary file ".pyenv/versions/3.14-dev/lib/libpython3.14.so.1.0", at Py_RunMain+0x996 [0x7858cb8f64e6]
  <truncated rest of calls>

Extension modules: pywatchman.bser, markupsafe._speedups (total: 2)
Segmentation fault (core dumped)

It also crashes on the current main branch of cpython (reproduced at 1550c30 compiled with debug):

$ runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
./Include/refcount.h:526: _Py_NegativeRefcount: Assertion failed: object has negative ref count
<object at 0x7fc6598b3930 is freed>
Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: initialized
Traceback (most recent call last):
  File "/cpython/Lib/unittest/suite.py", line 166, in _handleClassSetUp
    setUpClass()
    ~~~~~~~~~~^^
  File "/django/tests/test_runner/test_parallel.py", line 74, in setUpClass
    raise ValueError("woops")
ValueError: woops

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/django/tests/.test-python-dev/lib/python3.14/site-packages/tblib/pickling_support.py", line 30, in unpickle_exception
    inst.__traceback__ = tb
    ^^^^^^^^^^^^^^^^^^
TypeError: __traceback__ must be a traceback or None

Extension modules: pywatchman.bser, markupsafe._speedups (total: 2)
Aborted (core dumped)

I bisected this regression to the 053c285. Previously it passed with an assertion warning

./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/workspace/repo/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.002s

OK
Objects/typeobject.c:6226: _Py_NegativeRefcount: Assertion failed: object has negative ref count
Enable tracemalloc to get the memory block allocation traceback

object address  : 0x7cd1aa2aa1c0
object refcount : 4294967295
object type     : 0x63d9c0393140
object type name: type
object repr     : <class 'bserobj_tuple'>

Fatal Python error: _PyObject_AssertFailed: _PyObject_AssertFailed
Python runtime state: finalizing (tstate=0x000063d9c0404558)

Current thread 0x00007cd1ab58b740 (most recent call first):
  <no Python frame>
Aborted (core dumped)

CPython versions tested on:

3.14

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a7+ (heads/main:6a9bfee4fa1, Apr 25 2025, 10:31:59) [GCC 11.4.0]
@felixxm felixxm added the type-crash A hard crash of the interpreter, possibly with a core dump label May 3, 2025
@StanFromIreland
Copy link
Contributor

Can you please link the test code?

@felixxm
Copy link
Contributor Author

felixxm commented May 3, 2025

Can you please link the test code?

Added link to the issue description.

@ZeroIntensity
Copy link
Member

Looking at the faulthandler output, it looks like there are two extensions active, so it's possible this isn't our fault. (There have been some recent reference counting changes; if someone was incorrectly relying on some implementation details, then crashing makes sense.)

Would you mind isolating this down to something using the C API, or at least some Python code outside of a test suite?

@ZeroIntensity ZeroIntensity added the 3.14 bugs and security fixes label May 3, 2025
@felixxm
Copy link
Contributor Author

felixxm commented May 3, 2025

Looking at the faulthandler output, it looks like there are two extensions active, so it's possible this isn't our fault. (There have been some recent reference counting changes; if someone was incorrectly relying on some implementation details, then crashing makes sense.)

I uninstalled both packages (pywatchman and markupsafe) but the crash still occurs.

Would you mind isolating this down to something using the C API, or at least some Python code outside of a test suite?

Will try 🕵

@sobolevn
Copy link
Member

sobolevn commented May 3, 2025

cc @mpage

@mpage
Copy link
Contributor

mpage commented May 4, 2025

Will take a look on monday

@picnixz picnixz added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label May 4, 2025
@mpage
Copy link
Contributor

mpage commented May 5, 2025

I think the root cause of this issue was fixed by gh-133383. I cannot reproduce the crash on main at f5b7847. I can reproduce the crash if I revert 78adb63 on top of f5b7847.

@felixxm
Copy link
Contributor Author

felixxm commented May 5, 2025

I think the root cause of this issue was fixed by gh-133383. I cannot reproduce the crash on main at f5b7847. I can reproduce the crash if I revert 78adb63 on top of f5b7847.

You're right, thanks 👍 It still reports Objects/typeobject.c:6216: _Py_NegativeRefcount: Assertion failed: object has negative ref count, but no longer crashes.

@mpage
Copy link
Contributor

mpage commented May 5, 2025 •
edited
Loading

Hmm ... I'm not able to reproduce the assertion failure on main (or alpha 6). I see output that looks like:

(django_a6) > ./runtests.py test_runner.test_parallel.ParallelTestSuiteTest.test_handle_add_error_before_first_test
Testing against Django installed in '/data/users/mpage/django/django'
Found 1 test(s).
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.003s

OK
(django_a6) > python -VV
Python 3.14.0a6 (tags/v3.14.0a6:77b2c933cab, May  5 2025, 14:12:02) [GCC 11.5.0 20240719 (Red Hat 11.5.0-5)]
(django_a6) >

Are you able to bisect the assertion failure?

@felixxm
Copy link
Contributor Author

felixxm commented May 5, 2025

Hmm ... I'm not able to reproduce the assertion failure on main (or alpha 6). I see output that looks like:

Maybe because I configure python with --with-pydebug 🤔

./configure --with-pydebug && make -j

Are you able to bisect the assertion failure?

Should be doable, give me 1-2 days.

@mpage
Copy link
Contributor

mpage commented May 5, 2025

Maybe because I configure python with --with-pydebug

Hrm, I'm also compiling with --with-pydebug...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.14 bugs and security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@mpage @felixxm @sobolevn @picnixz @ZeroIntensity @StanFromIreland