email package should work better with unicode

[warsaw]

BPO	1685453
Nosy	@warsaw, @devdanzin, @merwok, @bitdancer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/bitdancer'
closed_at = <Date 2010-12-27.17:04:58.053>
created_at = <Date 2007-03-21.18:39:03.000>
labels = ['type-bug', 'library', 'expert-unicode']
title = 'email package should work better with unicode'
updated_at = <Date 2011-12-05.18:14:12.309>
user = 'https://github.com/warsaw'

bugs.python.org fields:

activity = <Date 2011-12-05.18:14:12.309>
actor = 'r.david.murray'
assignee = 'r.david.murray'
closed = True
closed_date = <Date 2010-12-27.17:04:58.053>
closer = 'r.david.murray'
components = ['Library (Lib)', 'Unicode']
creation = <Date 2007-03-21.18:39:03.000>
creator = 'barry'
dependencies = []
files = []
hgrepos = []
issue_num = 1685453
keywords = []
message_count = 7.0
messages = ['31612', '84700', '84753', '100550', '124715', '148880', '148882']
nosy_count = 9.0
nosy_names = ['barry', 'ocean-city', 'ajaksu2', 'eric.araujo', 'r.david.murray', 'bgamari', 'l0nwlf', 'pebbe', 'sivang']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1685453'
versions = ['Python 2.6', 'Python 3.1', 'Python 2.7', 'Python 3.2']

[warsaw]

This is a catch-all issue for improving the email package's handling of unicode. For now, please add issues/problems you find with email & unicode to this tracker item.

For example:

MIMEText()'s first argument should accept a unicode if _charset is also given. It should not be necessary to manually encode the first argument into an 8-bit string.

[devdanzin]

Link to bpo-1681333, bpo-4487, bpo-1443875, bpo-1555842, bpo-4661, bpo-1078919, bpo-963906,
bpo-1379416 and bpo-1368247.

[ocean-city]

Probably these are related too. bpo-5259 bpo-5304

[pebbe]

In Python 3.1.1, email.mime.text.MIMEText accepts an 8-bit charset, but not utf-8.

I think you should not have to specify a charset. All strings are unicode now, so I think the package should choose an appropriate charset based on the characters in the text, us-ascii, some iso-8859 charset, or utf-8, whatever fits.

Python 3.1.1 (r311:74480, Oct  2 2009, 11:50:52)                                                                                              
[GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)] on linux2                                                                                 
Type "help", "copyright", "credits" or "license" for more information.                                                                        
>>> from email.mime.text import MIMEText                                                                                                      
>>> text = 'H\u00e9'                                                                                                                          
>>> msg = MIMEText(text, 'plain', 'iso-8859-1')                                                                                               
>>> print(msg.as_string())                                                                                                                    
Content-Type: text/plain; charset="iso-8859-1"                                                                                                
MIME-Version: 1.0                                                                                                                             
Content-Transfer-Encoding: quoted-printable                                                                                                   
                                                                                                                                              
H=E9                                                                                                                                          
>>> msg = MIMEText(text, 'plain', 'utf-8')                                                                                                    
Traceback (most recent call last):                                                                                                            
  File "/my/opt/Python-3/lib/python3.1/email/message.py", line 269, in set_charset                                                            
    cte(self)                                                                                                                                 
TypeError: 'str' object is not callable                                                                                                       
                                                                                                                                              
During handling of the above exception, another exception occurred:                                                                           
                                                                                                                                              
Traceback (most recent call last):                                                                                                            
  File "<stdin>", line 1, in <module>                                                                                                         
  File "/my/opt/Python-3/lib/python3.1/email/mime/text.py", line 30, in __init__                                                              
    self.set_payload(_text, _charset)                                                                                                         
  File "/my/opt/Python-3/lib/python3.1/email/message.py", line 234, in set_payload                                                            
    self.set_charset(charset)                                                                                                                 
  File "/my/opt/Python-3/lib/python3.1/email/message.py", line 271, in set_charset                                                            
    self._payload = charset.body_encode(self._payload)                                                                                        
  File "/my/opt/Python-3/lib/python3.1/email/charset.py", line 380, in body_encode                                                            
    return email.base64mime.body_encode(string)                                                                                               
  File "/my/opt/Python-3/lib/python3.1/email/base64mime.py", line 94, in body_encode                                                          
    enc = b2a_base64(s[i:i + max_unencoded]).decode("ascii")                                                                                  
TypeError: must be bytes or buffer, not str                                                                                                   
>>>

[bitdancer]

Now that we are primarily focused on Python3 development, collecting "unicode" issues is not really all that useful (at least not to me, and I'm currently doing the email maintenance), so I'm closing this. All the relevant issues are assigned to me anyway, so I'll be dealing with them by and by.