And yes, it's not really a Python issue, it's a more generic issue with treating bytearrays as strings (what you refer to as POSIX text model). Until UTF-8 became the standard encoding everywhere, there were many Linux apps that couldn't handle non-Latin1 encodings properly, either. It was also the case on Windows in 9x days, for all the same reasons.
Most python2 code doesn't. It has a nice property of exploding spectacularly when the string you are writing out to console or file contains Chávez or Çelik.
Citation needed. I can't say I ever had practical issues with not being able to handle unicode on Python if I needed. No matter which library. Please give practical examples of where you cannot deal with unicode on Python 2.
for line in yourfile:
print line
</s>
Works fine on Linux.
I'm sorry to say, but your code for example. While I love your click library it has many small issues. Which annoy the heck out of me.
I would submit a patch but this is not a simple change and requires a bit of effort. Perhaps I'll find some time to work on it, but I'm afraid it might possibly break compatibility.
class ClickException(Exception):
"""An exception that Click can handle and show to the user."""
#: The exit code for this exception
exit_code = 1
def __init__(self, message):
ctor_msg = message
if PY2:
if ctor_msg is not None:
ctor_msg = ctor_msg.encode('utf-8')
Exception.__init__(self, ctor_msg)
self.message = message
def format_message(self):
return self.message
def __unicode__(self):
return self.message
def __str__(self):
return self.message.encode('utf-8')
def show(self, file=None):
if file is None:
file = get_text_stderr()
echo('Error: %s' % self.format_message(), file=file)
In python 3 you pass the message as is (it might cause another issue, but about that later).
In python 2 you immediately encode the passed variable using utf-8. This means that you're expecting argument to be of unicode type, but at the same time you're discouraging users to use unicode_literals, and in most situations users will pass a regular string.
In python 3 __str__ is trying to convert text to bytes, while the message would already be text. This most of the times will look correct, but it might spew garbage when there are non ascii characters.
Here's corrected code (did not test it though):
if PY2:
unicode = str
class ClickException(Exception):
"""An exception that Click can handle and show to the user."""
#: The exit code for this exception
exit_code = 1
def __init__(self, message):
if PY2 and isinstance(self, unicode):
message = message.encode()
Exception.__init__(self, message)
self.message = message
def format_message(self):
return self.message
def __unicode__(self):
return str(self).decode()
def __str__(self):
return str(self.message)
def show(self, file=None):
if file is None:
file = get_text_stderr()
echo('Error: %s' % self.format_message(), file=file)
In python 3 for example click refuses to run if LANG and LC_CTYPE are not defined. Why doesn't it simply do what all other applications are doing and simply fall back to latin-1 (ISO-8859-1) instead of printing the error.
Also, another issue (and my above code is also is affected by this). Click is checking for environmental variables before continuing, yet encode and decode have hardcoded utf-8. It probably should use whatever locale.getdefaultlocale() returns.
For some locales Windows has additional bonus, the Ansi (GUI) encoding is different than OEM (console) and it was heroic undertaking to make your program work correctly for both.
Python3 works out of the box.
For true insanity try adding IDLE to the mix. IDLE under Windows in Python 2 will let you output UTF-8, but won't accept it as input. Python 3 is more consistent.
I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I prefer Linux. At this point in time see no reason to cater to a shrinking server platform.
Not to mention they didn't even get unicode right with Python3. Google did with Go (assume UTF8).