UnicodeEncodeError: surrogates not allowed when parsing invalid charset

[andersk]

BPO	43323
Nosy	@warsaw, @bitdancer, @andersk, @serhiy-storchaka, @glaubitz
PRs	bpo-43323: Fix UnicodeEncodeError in the email module #32137

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-25.18:18:08.224>
labels = ['type-bug', 'expert-email', '3.10', '3.11', 'library', '3.9']
title = 'UnicodeEncodeError: surrogates not allowed when parsing invalid charset'
updated_at = <Date 2022-03-28.22:29:32.019>
user = 'https://github.com/andersk'

bugs.python.org fields:

activity = <Date 2022-03-28.22:29:32.019>
actor = 'glaubitz'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)', 'email']
creation = <Date 2021-02-25.18:18:08.224>
creator = 'andersk'
dependencies = []
files = []
hgrepos = []
issue_num = 43323
keywords = ['patch']
message_count = 9.0
messages = ['387685', '410426', '416092', '416101', '416107', '416109', '416112', '416113', '416225']
nosy_count = 6.0
nosy_names = ['barry', 'r.david.murray', 'andersk', 'serhiy.storchaka', 'mdengler', 'glaubitz']
pr_nums = ['32137']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43323'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[andersk]

We ran into a UnicodeEncodeError exception using email.parser to parse this email <https://lists.cam.ac.uk/pipermail/cl-isabelle-users/2021-February/msg00135.html\>, with full headers available in the raw archive <https://lists.cam.ac.uk/pipermail/cl-isabelle-users/2021-February.txt\>. The offending header is hilariously invalid:

Content-Type: text/plain; charset*=utf-8”''utf-8%E2%80%9D

but I’m filing an issue since the parser is intended to be robust against invalid input. Minimal reproduction:

>>> import email, email.policy
>>> email.message_from_bytes(b"Content-Type: text/plain; charset*=utf-8\xE2\x80\x9D''utf-8%E2%80%9D", policy=email.policy.default)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/email/__init__.py", line 46, in message_from_bytes
    return BytesParser(*args, **kws).parsebytes(s)
  File "/usr/local/lib/python3.10/email/parser.py", line 123, in parsebytes
    return self.parser.parsestr(text, headersonly)
  File "/usr/local/lib/python3.10/email/parser.py", line 67, in parsestr
    return self.parse(StringIO(text), headersonly=headersonly)
  File "/usr/local/lib/python3.10/email/parser.py", line 57, in parse
    return feedparser.close()
  File "/usr/local/lib/python3.10/email/feedparser.py", line 187, in close
    self._call_parse()
  File "/usr/local/lib/python3.10/email/feedparser.py", line 180, in _call_parse
    self._parse()
  File "/usr/local/lib/python3.10/email/feedparser.py", line 256, in _parsegen
    if self._cur.get_content_type() == 'message/delivery-status':
  File "/usr/local/lib/python3.10/email/message.py", line 578, in get_content_type
    value = self.get('content-type', missing)
  File "/usr/local/lib/python3.10/email/message.py", line 471, in get
    return self.policy.header_fetch_parse(k, v)
  File "/usr/local/lib/python3.10/email/policy.py", line 163, in header_fetch_parse
    return self.header_factory(name, value)
  File "/usr/local/lib/python3.10/email/headerregistry.py", line 608, in __call__
    return self[name](name, value)
  File "/usr/local/lib/python3.10/email/headerregistry.py", line 196, in __new__
    cls.parse(value, kwds)
  File "/usr/local/lib/python3.10/email/headerregistry.py", line 453, in parse
    kwds['decoded'] = str(parse_tree)
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 126, in __str__
    return ''.join(str(x) for x in self)
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 126, in <genexpr>
    return ''.join(str(x) for x in self)
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 798, in __str__
    for name, value in self.params:
  File "/usr/local/lib/python3.10/email/_header_value_parser.py", line 783, in params
    value = value.decode(charset, 'surrogateescape')
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 5-7: surrogates not allowed