3.11 regression: traceback.format_list raises UnicodeDecodeError in certain scenarios

[sebastinas]

Bug report

Take the following piece of code:

import sys, traceback
try:
    ｗｉｄｔｈ
except:
    _, _, tb = sys.exc_info()

tblist = traceback.extract_tb(tb)
print(traceback.format_list(tblist))

With Python 3.10 and earlier versions, executing a file with this code produces:

['  File "/tmp/test.py", line 3, in <module>\n    ｗｉｄｔｈ\n']

With Python 3.11, a UnicodeDecodeError is raised instead:

Traceback (most recent call last):
  File "/tmp/test.py", line 9, in <module>
    print(traceback.format_list(tblist))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/traceback.py", line 41, in format_list
    return StackSummary.from_list(extracted_list).format()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/traceback.py", line 531, in format
    formatted_frame = self.format_frame_summary(frame_summary)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/traceback.py", line 478, in format_frame_summary
    colno = _byte_offset_to_character_offset(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/traceback.py", line 566, in _byte_offset_to_character_offset
    return len(as_utf8[:offset + 1].decode("utf-8"))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xef in position 4: unexpected end of data

Your environment

• CPython versions tested on: 3.10.8 and 3.11.0
• Operating system and architecture: Debian unstable, amd64

[sebastinas]

Current main (d578aae) also has the same issue.

[serhiy-storchaka]

cc @ammaraskar, @pablogsal, @isidentical

[mdboom]

Also cc @iritkatriel (related to source positions -- in this case a source position in the middle of a UTF-8 codepoint).

[isidentical]

It seems like we are not mirroring the behaviour from the C implementation, which uses errors="replace" instead of the default strict. I'll try to send a PR soon. Thanks for the report @sebastinas!

[mdboom]

@isidentical: I was just looking at this as well, and there is also an off-by-one difference between this and the C implementation.

[isidentical]

@mdboom you are right! That is also an issue (in the C Implementation, we do a lengths check to not to overflow; although it is not necessary here. Another problem is the caret positions which actually might differ from node's offsets). Both should now be resolved with #98824. Thanks for catching it.

[isidentical]

Should be fixed now, thanks everyone!