Weird race condition exception / segfault in combination with Python 3.11

[Themanwithoutaplan]

Last week we started seeing segfaults when running CI for openpyxl against Python 3.11 (Docker). The problem was reproducible on my MacBook but only when running all tests, ie. tox -e py311 openpyxl or pytest -rf openpyxl but not when running the test in question or for the folder. The segfault seems to have gone away in recent tests but the exception is weird.

The original failure with segfault can be seen here: https://foss.heptapod.net/openpyxl/openpyxl/-/jobs/618571

• a detailed description of the bug or problem you are having
  exc = <class 'ResourceWarning'>
  value = ResourceWarning("unclosed file <_io.BufferedReader name='sample.xlsx'>")
  tb = <traceback object at 0x10e3f4b00>, limit = None, chain = True

  def format_exception(exc, /, value=_sentinel, tb=_sentinel, limit=None,
  chain=True):
  """Format a stack trace and the exception information.

    The arguments have the same meaning as the corresponding arguments
    to print_exception().  The return value is a list of strings, each
    ending in a newline and some containing internal newlines.  When
    these lines are concatenated and printed, exactly the same text is
    printed as does print_exception().
    """
    value, tb = _parse_value_tb(exc, value, tb)
  

  te = TracebackException(type(value), value, tb, limit=limit, compact=True)

chain = True
exc = <class 'ResourceWarning'>
limit = None
tb = <traceback object at 0x10e3f4b00>
value = ResourceWarning("unclosed file <_io.BufferedReader name='sample.xlsx'>")

/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:139:

---

/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:688: in init
self.stack = StackSummary._extract_from_extended_frame_gen(
_seen = {4539126048}
capture_locals = False
compact = True
exc_traceback = <traceback object at 0x10e3f4b00>
exc_type = <class 'ResourceWarning'>
exc_value = ResourceWarning("unclosed file <_io.BufferedReader name='sample.xlsx'>")
is_recursive_call = False
limit = None
lookup_lines = True
max_group_depth = 10
max_group_width = 15
self = <traceback.TracebackException object at 0x10e4029d0>
/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:416: in _extract_from_extended_frame_gen
for f, (lineno, end_lineno, colno, end_colno) in frame_gen:
capture_locals = False
fnames = set()
frame_gen = <generator object _walk_tb_with_full_positions at 0x10d50c3c0>
klass = <class 'traceback.StackSummary'>
limit = None
lookup_lines = True
result = []

---

tb = <traceback object at 0x10e3f4b00>

def _walk_tb_with_full_positions(tb):
    # Internal version of walk_tb that yields full code positions including
    # end line and column information.
    while tb is not None:

      positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)

E AttributeError: 'frozenset' object has no attribute 'f_code'

tb = <traceback object at 0x10e3f4b00>

/opt/local/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/traceback.py:353: AttributeError
========================================== short test summary info ==========================================
FAILED openpyxl/tests/test_iter.py::test_read_single_cell_formula[False-='Sheet2 - Numbers'!D5] - Attribut...
=========================== 1 failed, 2467 passed, 16 skipped, 6 xfailed in 8.24s ===========================
ERROR: InvocationError for command /Users/charlieclark/Projects/openpyxl/.tox/py311/bin/pytest openpyxl (exited with code 1)
__________________________________________________ summary __________________________________________________
ERROR: py311: commands failed

• [x ] output of pip list from the virtual environment you are using
  Package Version Editable project location

---

attrs 22.1.0
et-xmlfile 1.1.0
iniconfig 1.1.1
lxml 4.9.1
openpyxl 3.1.0b1 /Users/charlieclark/temp/openpyxl
packaging 21.3
Pillow 9.2.0
pip 22.2
pluggy 1.0.0
py 1.11.0
pyparsing 3.0.9
pytest 7.1.2
setuptools 63.2.0
tomli 2.0.1

• pytest and operating system versions
  pytest 7.1.2 and Python 3.11 beta 4 and 5
• minimal example if possible
  This is not possible. It seems some kind of race condition. Also reported on Weird test failures related to tracebacks python/cpython#95818

[iritkatriel]

The original failure with segfault can be seen here: https://foss.heptapod.net/openpyxl/openpyxl/-/jobs/618571

The failure at that link is not a segfault, but rather an exception due to a tb.tb_frame which is of type frozenset:

>           positions = _get_code_position(tb.tb_frame.f_code, tb.tb_lasti)
[212](https://foss.heptapod.net/openpyxl/openpyxl/-/jobs/618571#L212)E           AttributeError: 'frozenset' object has no attribute 'f_code'
[213](https://foss.heptapod.net/openpyxl/openpyxl/-/jobs/618571#L213)tb         = <traceback object at 0x7f249c6c1f80>

Is pytest doing anything that could have created this situation? Perhaps related to this?

[RonnyPfannschmidt]

At first glance this looks like a issue with stale pointers
, a frame replaced with a frozen set indicates a free + reuse error

Debugging that will be hard

[iritkatriel]

@RonnyPfannschmidt Are you able to reproduce it? I'd start by finding out what's in that frozenset (by monkey patching as in this link, or with a breakpoint).

[RonnyPfannschmidt]

@iritkatriel i won't be able to dig into this anytime soon, i strongly suspect the issue is at the c level away from pytest

[iritkatriel]

You can close this. We have a reproducer now.

[RonnyPfannschmidt]

Thanks, closing as python issue