Skip to content

Segmentation fault in PGO tests with free-threading build #122921

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
furkanonder opened this issue Aug 11, 2024 · 1 comment
Closed

Segmentation fault in PGO tests with free-threading build #122921

furkanonder opened this issue Aug 11, 2024 · 1 comment
Labels
3.14 bugs and security fixes tests Tests in the Lib/test dir topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump

Comments

@furkanonder
Copy link
Contributor

furkanonder commented Aug 11, 2024 •
edited by github-actions bot
Loading

Crash report

What happened?

Configuration:

./configure --enable-optimizations --with-trace-refs --disable-gil

configure-output.txt

$ make -j

Output:

./python -E -c 'import sys ; from sysconfig import get_platform ; print("%s-%d.%d" % (get_platform(), *sys.version_info[:2]))' >platform
Checked 112 modules (34 built-in, 77 shared, 1 n/a on linux-x86_64, 0 disabled, 0 missing, 0 failed on import)
make[2]: Leaving directory '/home/arf/cpython'
touch profile-gen-stamp
make[1]: Leaving directory '/home/arf/cpython'
# Next, run the profile task to generate the profile information.
./python -m test --pgo --timeout=
Using random seed: 3385972810
0:00:00 load avg: 41.36 Run 44 tests sequentially in a single process
0:00:00 load avg: 41.36 [ 1/44] test_array
0:00:01 load avg: 41.36 [ 2/44] test_base64
0:00:01 load avg: 41.36 [ 3/44] test_binascii
0:00:01 load avg: 41.36 [ 4/44] test_binop
0:00:01 load avg: 41.36 [ 5/44] test_bisect
0:00:01 load avg: 41.36 [ 6/44] test_bytes
0:00:07 load avg: 38.13 [ 7/44] test_bz2
0:00:08 load avg: 35.32 [ 8/44] test_cmath
0:00:08 load avg: 35.32 [ 9/44] test_codecs
0:00:10 load avg: 35.32 [10/44] test_collections
0:00:12 load avg: 32.57 [11/44] test_complex
0:00:13 load avg: 32.57 [12/44] test_dataclasses
0:00:14 load avg: 32.57 [13/44] test_datetime
0:00:20 load avg: 30.12 [14/44] test_decimal
0:00:26 load avg: 27.87 [15/44] test_difflib
0:00:28 load avg: 25.80 [16/44] test_embed
0:00:40 load avg: 22.14 [17/44] test_float
0:00:40 load avg: 22.14 [18/44] test_fstring
0:00:42 load avg: 22.14 [19/44] test_functools
0:00:43 load avg: 20.61 [20/44] test_generators
0:00:44 load avg: 20.61 [21/44] test_hashlib
Fatal Python error: Segmentation fault

Thread 0x00007a017ee006c0 (most recent call first):
  File "/home/arf/cpython/Lib/test/test_hashlib.py", line 947 in hash_in_chunks
  File "/home/arf/cpython/Lib/threading.py", line 992 in run
make: *** [Makefile:907: profile-run-stamp] Segmentation fault (core dumped)

make-output.txt

After removing --disable-gil

./configure --enable-optimizations --with-trace-refs && make -j

Output:

make[1]: Leaving directory '/home/arf/cpython'
# Next, run the profile task to generate the profile information.
./python -m test --pgo --timeout=
Using random seed: 3313315501
0:00:00 load avg: 35.47 Run 44 tests sequentially in a single process
0:00:00 load avg: 35.47 [ 1/44] test_array
0:00:00 load avg: 32.71 [ 2/44] test_base64
0:00:01 load avg: 32.71 [ 3/44] test_binascii
0:00:01 load avg: 32.71 [ 4/44] test_binop
0:00:01 load avg: 32.71 [ 5/44] test_bisect
0:00:01 load avg: 32.71 [ 6/44] test_bytes
0:00:04 load avg: 32.71 [ 7/44] test_bz2
0:00:05 load avg: 30.25 [ 8/44] test_cmath
0:00:05 load avg: 30.25 [ 9/44] test_codecs
0:00:06 load avg: 30.25 [10/44] test_collections
0:00:08 load avg: 30.25 [11/44] test_complex
0:00:08 load avg: 30.25 [12/44] test_dataclasses
0:00:08 load avg: 30.25 [13/44] test_datetime
0:00:13 load avg: 27.83 [14/44] test_decimal
0:00:17 load avg: 25.76 [15/44] test_difflib
0:00:18 load avg: 25.76 [16/44] test_embed
0:00:29 load avg: 22.19 [17/44] test_float
0:00:29 load avg: 22.19 [18/44] test_fstring
0:00:30 load avg: 20.74 [19/44] test_functools
0:00:31 load avg: 20.74 [20/44] test_generators
0:00:31 load avg: 20.74 [21/44] test_hashlib
0:00:31 load avg: 20.74 [22/44] test_heapq
0:00:32 load avg: 20.74 [23/44] test_int
0:00:33 load avg: 20.74 [24/44] test_itertools
0:00:36 load avg: 19.32 [25/44] test_json
0:00:39 load avg: 19.32 [26/44] test_long
0:00:42 load avg: 17.93 [27/44] test_lzma
0:00:42 load avg: 17.93 [28/44] test_math
0:00:45 load avg: 16.65 [29/44] test_memoryview
0:00:45 load avg: 16.65 [30/44] test_operator
0:00:45 load avg: 16.65 [31/44] test_ordered_dict
0:00:46 load avg: 16.65 [32/44] test_patma
0:00:47 load avg: 16.65 [33/44] test_pickle
0:00:52 load avg: 15.48 [34/44] test_pprint
0:00:53 load avg: 15.48 [35/44] test_re
0:00:53 load avg: 15.48 [36/44] test_set
0:00:58 load avg: 14.32 [37/44] test_sqlite3
0:00:59 load avg: 14.32 [38/44] test_statistics
0:01:07 load avg: 12.27 [39/44] test_str
0:01:10 load avg: 12.27 [40/44] test_struct
0:01:10 load avg: 11.37 [41/44] test_tabnanny
0:01:11 load avg: 11.37 [42/44] test_time
0:01:13 load avg: 11.37 [43/44] test_xml_etree
0:01:14 load avg: 11.37 [44/44] test_xml_etree_c

Total duration: 1 min 15 sec
Total tests: run=9,346 skipped=205
Total test files: run=44/44
Result: SUCCESS

In this case the PGO tests pass, but the asyncio test fails.

╰─$ ./python -m test -j12
== CPython 3.14.0a0 (heads/main:1069190bad, Aug 12 2024, 01:52:17) [GCC 14.2.1 20240805]
== Linux-6.10.3-arch1-2-x86_64-with-glibc2.40 little-endian
== Python build: release PGO TraceRefs
== cwd: /home/arf/cpython/build/test_python_worker_647363æ
== CPU count: 16
== encodings: locale=UTF-8 FS=utf-8
== resources: all test resources are disabled, use -u option to unskip tests

Using random seed: 1371718611
0:00:00 load avg: 29.25 Run 479 tests in parallel using 12 worker processes
0:00:00 load avg: 29.25 [  1/479] test.test_asyncio.test_buffered_proto passed
0:00:00 load avg: 29.25 [  2/479] test.test_asyncio.test_protocols passed
0:00:00 load avg: 29.25 [  3/479] test.test_asyncio.test_futures2 passed
0:00:00 load avg: 29.25 [  4/479] test.test_asyncio.test_context passed
0:00:00 load avg: 29.25 [  5/479] test.test_asyncio.test_proactor_events passed
0:00:00 load avg: 29.25 [  6/479] test.test_asyncio.test_pep492 passed
0:00:00 load avg: 29.25 [  7/479] test.test_asyncio.test_queues passed
0:00:00 load avg: 29.25 [  8/479] test.test_asyncio.test_selector_events passed
0:00:00 load avg: 29.25 [  9/479] test.test_asyncio.test_runners passed
0:00:00 load avg: 29.25 [ 10/479/1] test.test_asyncio.test_server failed (1 failure)
test test.test_asyncio.test_server failed -- Traceback (most recent call last):
  File "/home/arf/cpython/Lib/unittest/async_case.py", line 93, in _callTestMethod
    result = self._callMaybeAsync(method)
  File "/home/arf/cpython/Lib/unittest/async_case.py", line 119, in _callMaybeAsync
    return self._asyncioRunner.run(
           ~~~~~~~~~~~~~~~~~~~~~~~^
        func(*args, **kwargs),
        ^^^^^^^^^^^^^^^^^^^^^^
        context=self._asyncioTestContext,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/arf/cpython/Lib/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/arf/cpython/Lib/asyncio/base_events.py", line 721, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/home/arf/cpython/Lib/test/test_asyncio/test_server.py", line 249, in test_abort_clients
    self.assertNotEqual(s_wr.transport.get_write_buffer_size(), 0)
    ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 0 == 0

After removing --with-trace-refs

╰─$ ./configure --enable-optimizations --disable-gil  && make -j

Output:

./python -m test --pgo --timeout=
Using random seed: 1327628949
0:00:00 load avg: 39.61 Run 44 tests sequentially in a single process
0:00:00 load avg: 39.61 [ 1/44] test_array
0:00:00 load avg: 36.68 [ 2/44] test_base64
0:00:01 load avg: 36.68 [ 3/44] test_binascii
0:00:01 load avg: 36.68 [ 4/44] test_binop
0:00:01 load avg: 36.68 [ 5/44] test_bisect
0:00:01 load avg: 36.68 [ 6/44] test_bytes
0:00:05 load avg: 33.98 [ 7/44] test_bz2
0:00:06 load avg: 33.98 [ 8/44] test_cmath
0:00:06 load avg: 33.98 [ 9/44] test_codecs
0:00:08 load avg: 33.98 [10/44] test_collections
0:00:10 load avg: 33.98 [11/44] test_complex
0:00:10 load avg: 33.98 [12/44] test_dataclasses
0:00:10 load avg: 31.50 [13/44] test_datetime
0:00:16 load avg: 29.14 [14/44] test_decimal
0:00:20 load avg: 26.88 [15/44] test_difflib
0:00:22 load avg: 26.88 [16/44] test_embed
0:00:31 load avg: 23.14 [17/44] test_float
0:00:32 load avg: 23.14 [18/44] test_fstring
0:00:33 load avg: 23.14 [19/44] test_functools
0:00:35 load avg: 23.14 [20/44] test_generators
0:00:35 load avg: 21.44 [21/44] test_hashlib
0:00:36 load avg: 21.44 [22/44] test_heapq
0:00:37 load avg: 21.44 [23/44] test_int
0:00:37 load avg: 21.44 [24/44] test_itertools
0:00:41 load avg: 20.13 [25/44] test_json
0:00:45 load avg: 20.13 [26/44] test_long
0:00:47 load avg: 18.84 [27/44] test_lzma
0:00:48 load avg: 18.84 [28/44] test_math
0:00:51 load avg: 17.49 [29/44] test_memoryview
0:00:52 load avg: 17.49 [30/44] test_operator
0:00:52 load avg: 17.49 [31/44] test_ordered_dict
0:00:55 load avg: 17.49 [32/44] test_patma
0:00:55 load avg: 16.89 [33/44] test_pickle
0:01:03 load avg: 15.78 [34/44] test_pprint
0:01:03 load avg: 15.78 [35/44] test_re
0:01:05 load avg: 15.78 [36/44] test_set
0:01:11 load avg: 14.27 [37/44] test_sqlite3
0:01:12 load avg: 14.27 [38/44] test_statistics
0:01:21 load avg: 12.53 [39/44] test_str
0:01:23 load avg: 12.53 [40/44] test_struct
0:01:24 load avg: 12.53 [41/44] test_tabnanny
0:01:25 load avg: 12.53 [42/44] test_time
0:01:27 load avg: 11.69 [43/44] test_xml_etree
0:01:29 load avg: 11.69 [44/44] test_xml_etree_c

Total duration: 1 min 32 sec
Total tests: run=9,346 skipped=206
Total test files: run=44/44
Result: SUCCESS
true

./python -m test -j12

Output:

== Tests result: SUCCESS ==

18 tests skipped:
    test.test_asyncio.test_windows_events
    test.test_asyncio.test_windows_utils test.test_gdb.test_backtrace
    test.test_gdb.test_cfunction test.test_gdb.test_cfunction_full
    test.test_gdb.test_misc test.test_gdb.test_pretty_print
    test_android test_devpoll test_interpreters test_kqueue
    test_launcher test_msvcrt test_startfile test_winapi
    test_winconsoleio test_winreg test_wmi

11 tests skipped (resource denied):
    test_curses test_peg_generator test_pyrepl test_smtpnet
    test_socketserver test_tkinter test_ttk test_urllib2net
    test_urllibnet test_winsound test_zipfile64

450 tests OK.

Total duration: 1 min 45 sec
Total tests: run=44,111 skipped=1,913
Total test files: run=468/479 skipped=18 resource_denied=11
Result: SUCCESS

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.14.0a0 experimental free-threading build (heads/main:1069190bad, Aug 12 2024, 01:46:15) [GCC 14.2.1 2024080]
@furkanonder furkanonder added tests Tests in the Lib/test dir type-crash A hard crash of the interpreter, possibly with a core dump topic-free-threading 3.14 bugs and security fixes labels Aug 11, 2024
@colesbury
Copy link
Contributor

--with-trace-refs is not thread-safe with the free-threaded build. I don't think this is related to PGO.

@ZeroIntensity ZeroIntensity mentioned this issue Sep 13, 2024
Closed
@mdboom mdboom mentioned this issue Nov 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
3.14 bugs and security fixes tests Tests in the Lib/test dir topic-free-threading type-crash A hard crash of the interpreter, possibly with a core dump
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@colesbury @furkanonder