email.parser: impossible to read messages encoded in a different encoding

[dato]

BPO	4661
Nosy	@warsaw, @pitrou, @vstinner, @bitdancer
Files	email_parse_bytes.diff email_parse_bytes2.diff email_parse_bytes3.diff email_parse_bytes4.diff: completed patch set email_parse_bytes5.diff: svn diff against current py3k trunk email_parse_bytes7.diff email_parse_bytes8.diff email_parse_bytes9.diff BytesParser_newline.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/bitdancer'
closed_at = <Date 2010-12-12.18:03:17.612>
created_at = <Date 2008-12-14.16:55:45.574>
labels = ['type-bug', 'expert-email']
title = 'email.parser: impossible to read messages encoded in a different encoding'
updated_at = <Date 2014-10-03.01:07:07.673>
user = 'https://bugs.python.org/dato'

bugs.python.org fields:

activity = <Date 2014-10-03.01:07:07.673>
actor = 'yu.zhao@getcwd.com'
assignee = 'r.david.murray'
closed = True
closed_date = <Date 2010-12-12.18:03:17.612>
closer = 'r.david.murray'
components = ['email']
creation = <Date 2008-12-14.16:55:45.574>
creator = 'dato'
dependencies = []
files = ['18962', '19078', '19102', '19110', '19113', '19155', '19160', '19161', '36788']
hgrepos = []
issue_num = 4661
keywords = ['patch']
message_count = 25.0
messages = ['77807', '89508', '89509', '91737', '95293', '117118', '117119', '117120', '117774', '117856', '117857', '117893', '117897', '117900', '118160', '118175', '118191', '118192', '118197', '118198', '118199', '118201', '118204', '123843', '228289']
nosy_count = 9.0
nosy_names = ['barry', 'anadelonbrin', 'pitrou', 'vstinner', 'r.david.murray', 'dato', 'tercero12', 'meatballhat', 'yu.zhao@getcwd.com']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4661'
versions = ['Python 3.2']

[dato]

Currently, email.parser/feedparser can only parse messages that come
as a string, or from a file opened in text mode.

Email messages, however, can contain 8bit characters in any encoding
other than the local one (yet still be valid e-mails, of course), so I
think a method is needed to have the parser be able to receive bytes.
At the moment, and as far as I can see, it's not possible to parse
some perfectly valid messages with python 3.0.

I don't think it's appropriate to ask that files be opened with the
proper encoding, and then for the parser to read them. First, it is
not possible to know what encoding that would be without parsing the
message. And second, a message could contain parts in different
encoding, and many mailboxes with random messages most certainly do.

Also, message objects will need a way to return a bytes repreentation,
for the reasons explained above, and particularly if one wants to
write back the message without modifying it.

[bitdancer]

Is there any use case for feedparser accepting strings as input that
isn't a design error waiting to bite the programmer?

[warsaw]

dato: We've started some branches that try to address this, by exposing
both a read-a-buncha-bytes interface and a read-a-string interface.

rdm: As it turns out, yes. There are use cases for reading a string
containing only ascii bytes.

In general, this is part of the big tricky problem in fixing the email
package for Python 3.x. We had great debates about this at Pycon with
no resolution, and I think everyone has just been too busy to engage on
this since then. :(

email-sig is the best place to try to rally some effort.

[AlexQuinn]

This bug also prevents the cgi module from handling POST data with
multipart/form-data. Consequently, 3.x cannot be readily used to write
web apps for uploading files. See bpo-4953:
http://bugs.python.org/issue4953

[tercero12]

Just an update for people interested:

The email team has a goal of fixing the email module in time for the 3.2
release. There is the possibility of having to change some interfaces.
See this document: http://wiki.python.org/moin/Email%20SIG/DesignThoughts

[bitdancer]

OK, I'm not entirely sure I want to post this, but....

Antoine and I were having a conversation about nntplib and email and I noted that unicode as an email transmission channel acts as if it required 7bit clean data. That is, that there's no way to use unicode as an 8bit data transmission channel. Antoine pointed out that there is PEP-383, and that he is using that in his nntplib update to tunnel 8bit data (if there is any) from and back to the nntp server. I said I couldn't do that with email because I not only needed to transmit the data, I also needed to *parse* it.

Antoine pointed out that you can in fact parse a header even if it has surrogateescape code points in it.

So I started thinking about that. In point of fact, from the point of view of an email parser, non-ASCII bytes are pretty much opaque. They don't affect the semantics of the parsing. Either they are invalid data (in headers), or they are opaque content data (8bit Content-Transfer-Encoding).

So...I came up with a horrible little hack, which is attached here as a patch. This is horrible because it is a perversion of the Python3 desire to make a clean separation between bytes and strings. The only thing it really has to recommend it is that it works: it allows email5 (the version of email currently in Python3) to read wire-format messages and parse them into valid message structures.

The patch is a proof of concept and is far from complete. It handles only message bodies (but those are the most important) and has no doc updates and only one test. If this approach is deemed worth considering, I will flesh out the tests and make sure the corner cases are handled correctly, and write docs with lots of notes about why this is perverse and email6 will make it all better :)

I feel bad about posting this both because it is an ugly hack and because it will likely slow down email6 development (because it will make email5 mostly work). But making email5 mostly work in 3.2 seems like a case where practicality beats purity.

The essence of the hack is as follows: Given binary data we encode it to ASCII using the surrogateescape error handler. Then, when a message body is retrieved we check to see if there are any surrogates in it, and if there are we encode it back to ASCII using surrogateescape, thereby recovering the original bytes. For "Content-Transfer-Encoding: 8bit" parts we can then try to decode it using the declared charset, or ASCII with the replace error handler if the charset isn't known. But in any case the original binary data is accessible by using 'decode=True' in the call to get_payload. (NB for those not familiar with the API: decode=True refers to decoding the Content-Transfer-Encoding, *not* decoding to unicode...which means after CTE decoding you end up with a byte string).

For headers, which are not supposed to have 8bit data in them, the best we can do is re-decode them with ASCII/replace, but at least it will be possible to parse the messages. (The current patch doesn't do this.)

Another thing missing from the current patch is the generator side. But since the binary data for the message content is now available, it should be possible to have a generator that outputs binary.

Note that in this patch I've introduced new functions/methods for getting binary string data in, but for file input one needs to open the file as text using ASCII encoding and the surrogateescape error handler.

I've only done minimal testing on this (obviously), and so I may find a showstopper somewhere along the way, but so far it seems to work, and logically it seems like it should work.

I don't know if that makes me happy or sad :)

[pitrou]

A couple of comments:

• what is str(self.get_param('charset', 'ascii')) supposed to achieve? does get_param() return a bytes object?
• instead of ascii+surrogateescape, you could simply use latin1

[bitdancer]

The 'str' around get_param shouldn't be there, that was left over from an earlier version of the patch.

I use surrogateescape rather than latin1 because using surrogateescape with ascii encoding gives me a reliable way to know whether or not the original source was bytes containing non-ascii chars. (If it was bytes containing only ascii chars, then it doesn't matter what the original source was.)

[bitdancer]

New version of the patch that adds many more tests, and handles non-ASCII bytes in header values by changing them to '?'s when the header value is retrieved as a string. I think I'm half done. Still to do: generate_bytes, and the doc updates.

By the way, another important reason to use surrogateescape rather than latin1 is that if I miss something and the byte-containing-strings escape, it will be obvious that that is what happened. Otherwise we're back in Python2 bytes/string conflation land.

I of course make no promises about performance. And there is an issue there in that every header value access is now wrapped in an additional function call and a regex test, at a minimum, whether there are bytes present in the input or not :(

[bitdancer]

New version of patch including a BytesGenerator.