mboxMessage.get_payload throws TypeError on malformed content type

[enrico]

BPO	36180
Nosy	@warsaw, @bitdancer, @mapreri, @tirkarthi
Files	broken.zip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2019-03-04.11:03:37.379>
labels = ['3.8', 'type-bug', '3.7', 'expert-email']
title = 'mboxMessage.get_payload throws TypeError on malformed content type'
updated_at = <Date 2019-04-14.06:40:20.881>
user = 'https://bugs.python.org/enrico'

bugs.python.org fields:

activity = <Date 2019-04-14.06:40:20.881>
actor = 'xtreak'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['email']
creation = <Date 2019-03-04.11:03:37.379>
creator = 'enrico'
dependencies = []
files = ['48184']
hgrepos = []
issue_num = 36180
keywords = []
message_count = 2.0
messages = ['337091', '340187']
nosy_count = 5.0
nosy_names = ['barry', 'r.david.murray', 'enrico', 'mapreri', 'xtreak']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'test needed'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue36180'
versions = ['Python 3.7', 'Python 3.8']

Linked PRs

• gh-80361: Fix TypeError in email.Message.get_payload() #117994
• [3.12] gh-80361: Fix TypeError in email.Message.get_payload() (GH-117994) #117998

[enrico]

This simple code:

import mailbox

mbox = mailbox.mbox("broken.mbox")
for msg in mbox:
    msg.get_payload()

Fails rather unexpectedly:

$ python3 broken.py 
Traceback (most recent call last):
  File "broken.py", line 5, in <module>
    msg.get_payload()
  File "/usr/lib/python3.7/email/message.py", line 267, in get_payload
    payload = bpayload.decode(self.get_param('charset', 'ascii'), 'replace')
TypeError: decode() argument 1 must be str, not tuple

(I'm attaching a zip with code and mailbox)

I would have expected either that the part past text/plain is ignored if it doesn't make sense, or that content-type is completely ignored.

I have to process a large mailbox archive, and this is currently how I had to work around this issue, and it's causing me to have to skip email content which would otherwise be reasonably accessible:

https://salsa.debian.org/nm-team/echelon/commit/617ce935a31f6256257ffb24e11a5666306406c3

[tirkarthi]

A simplified reproducer as below. The tuple is returned from here

cpython/Lib/email/message.py

Line 93 in 830b43d

def _unquotevalue(value):

and perhaps is an untested code path? The charset gets a tuple value of ('utf-8��', '', '"utf-8Â\xa0"') .

import mailbox
import tempfile

broken_message = """
From [[email protected]](mailto:[email protected]) Wed Sep 24 01:22:15 2003
Date: Wed, 24 Sep 2003 07:05:50 +0200
From: Test test <[[email protected]](mailto:[email protected])>
To: [[email protected]](mailto:[email protected])
Subject: Re: Test
Mime-Version: 1.0
Content-Type: text/plain; charset*=utf-8†''utf-8%C2%A0

trés intéressé
"""

with tempfile.NamedTemporaryFile() as f:
    f.write(broken_message.encode())
    f.seek(0)
    msg = mailbox.mbox(f.name)
    for m in msg:
        print(m.get_payload())

$ ../cpython/python.exe bpo36180.py
Traceback (most recent call last):
  File "bpo36180.py", line 21, in <module>
    print(m.get_payload())
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/email/message.py", line 267, in get_payload
    payload = bpayload.decode(self.get_param('charset', 'ascii'), 'replace')
TypeError: decode() argument 1 must be str, not tuple
sys:1: ResourceWarning: unclosed file <_io.BufferedRandom name='/var/folders/2b/mhgtnnpx4z943t4cc9yvw4qw0000gn/T/tmp4ddavb6g'>