This could have been done differently. I would have been tempted to store strings as UTF-8 in Python 3, but hide this from the user. Most of the string functions, including regular expressions, don't really random access a string; they're sequential and forward only. They could work on UTF-8. The ones that return a string index should return an opaque type (say, "StringPos") which has a reference to the string and a position in it. The string functions should accept that type as an index.

If someone applies "int()" to a string index, or performs arithmetic on it, or indexes a string with an actual integer, then it's necessary to scan the entire UTF-8 string and create an index of where each Unicode character starts. This should be a rare event. Especially if there's support for adding or subtracting small integers from a StringPos by walking the string forwards or backwards, rather than building the full index.

Examples:

   s = "This is a test" # Unicode string, internal represention UTF-8
   i = s.find("is")     # returns a StringPos, not an int.
                        # no need to index the string for any of these
   s1 = s[i]            # StringPos selects the char found by find
   s2 = s[i:]           # No need to build an index for this
   s3 = s[i+1:]         # Adding a small int just walks the string
                        # these force generation of an index 
                        # the first operation like this on a string is expensive
                        # comparable to a UTF-8 to rune array conversion
   j = int(i)           # converts a StringPos to an int 
   s4 = s[i*2]          # again, expensive

This would cut Python memory consumption way down when processing mostly-ASCII Unicode data. It would also make it more efficient to input and output UTF-8, which is usually what you want.

(Granted, I'm still learning, so maybe I can be as efficient without relying on unsafe, but still...)

This is incorrect for anything other than an ASCII-only world. In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character.

By rubbing your nose in the difference between str and bytes, Python 3 complicates your life.

Having had a lot of experience with it, I would instead say Python 3 makes you think about things and find bugs at the appropriate time -- up-front during the development process -- rather than at an inappropriate time, like 3AM on a Sunday when your pager goes off because you trusted the "helpful" but wrong way of doing things.

Only if people writing code are entirely oblivious to what UTF-8 is. The most basic feature of UTF-8 is that it's a variable-length encoding scheme. If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?

Because UTF-8 is the Unicode encoding that looks just like ASCII, so long as the characters stay within the set supported by ASCII, so it lets them say they're doing "Unicode" while really just writing code to handle ASCII.

Using byte strings also doesn't 'just work' if the original format is a Unicode encoding other than UTF-8; e.g. Windows filenames are UTF-16 and can't be converted to UTF-8 without risking errors caused by unpaired surrogates. But that's what WTF-8 is for (or rather, what it should be for, disclaimers in the spec notwithstanding)...

I'm tired of people spreading FUD about it. they either don't understand what they are doing (and do it incorrectly) or are repeating what other people (who did things incorrectly) said:

    » echo $LANG
    en_US.UTF-8
    » echo $LC_CTYPE
    en_US.UTF-8
    » echo 'works' > $'\377'$'\377'$'\377'
    » ls
    ???
    » cat $'\377'$'\377'$'\377'
    works
    » python3
    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> a = b'\377\377\377'
    >>> a
    b'\xff\xff\xff'
    >>> a.decode()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
    >>> b = open(a)
    >>> b.read()
    'works\n'
    >>>

Python3 on the other side is strict about to point errors in your code. If python3 code will crash it's either bug in python or (most likely) bug in your code and you should fix it.

I (and I'm sure a lot of people) prefer program to crash, pointing where the error is, than silently corrupt data once in a while, and when you notice the issue spending days or months figuring out where the corruption is happening.

On the other hand, if you read a list of filenames from a file, you'd better remember to either use binary or explicitly choose surrogateescape decoding. Which is a special case of "file in a text-based format" from my original post.

I don't know what you mean by "corruption".

As for your other point. You're right, but there's no easy way resolving this. Either treat everything as unicode and remember it's binary on linux, or have linux working and everything else broken.

In that scenario you probably should use .encode('utf-8', 'surrogateescape') as soon as you can and work on bytes, so no need of doing the above.

Also, if you for example call os.listdir() with argument in bytes it'll return entries in bytes:

    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> os.listdir(b'.')
    [b'\xff\xff\xff', b'test.py']

Currently looks like you would have to use:

    filename.encode(sys.getfilesystemencoding(), 'surrogateescape').decode('iso-8859-1')

Let me guess: you don't work on anything that has any internationalization ever.

Even if we accept your myopic list of features that work and believe your statement that "most operations can be performed on bytes", that doesn't make the language usable for internationalized content. Trading having to call encode and decode, which fail with clear, easy-to-fix errors, for a myriad of subtle, silent encoding failures is not a good trade. Which you would be abundantly clear to you if you had internationalized anything ever.

After a while I started creatively expressing my dislike of phone screens by showing them an implementation that hints at some of the complexity lurking in this allegedly-simple task (i.e., you need to understand Unicode classes and normalization in order to write a palindrome tester).

I think if you showed me an implementation that hinted at some of the complexity lurking in the task, that would be a big point in your favor.

I did bust out a version on one call which did detection of combining classes, normalized into composed form before checking the reverse, etc., and the interviewer seemed kinda lost as I explained it.

Ended up not working for them, but for different reasons.

I question some details in your lists but imagine you'd agree that the Works vs Doesn't split approximates to whole string level vs sub-string/character level operations.

Python3 is the Windows way of handling strings so the change was pure churn. Both are valid, but Linux is "pretty popular" right? So why break everyone's code just to churn the pot.

If it was good for Kernigan and Richie, good for Joe Ossanna, good for Robert Pike well it's good enough for me.

I'll take Python2's POSIX text model over Microsoft's default unicode strings every time because I prefer Linux. At this point in time see no reason to cater to a shrinking server platform.

Not to mention they didn't even get unicode right with Python3. Google did with Go (assume UTF8).

And yes, it's not really a Python issue, it's a more generic issue with treating bytearrays as strings (what you refer to as POSIX text model). Until UTF-8 became the standard encoding everywhere, there were many Linux apps that couldn't handle non-Latin1 encodings properly, either. It was also the case on Windows in 9x days, for all the same reasons.

Most python2 code doesn't. It has a nice property of exploding spectacularly when the string you are writing out to console or file contains Chávez or Çelik.

Citation needed. I can't say I ever had practical issues with not being able to handle unicode on Python if I needed. No matter which library. Please give practical examples of where you cannot deal with unicode on Python 2.

    for line in yourfile:
        print line

</s>

Works fine on Linux.

I'm sorry to say, but your code for example. While I love your click library it has many small issues. Which annoy the heck out of me.

I would submit a patch but this is not a simple change and requires a bit of effort. Perhaps I'll find some time to work on it, but I'm afraid it might possibly break compatibility.

    class ClickException(Exception):
        """An exception that Click can handle and show to the user."""

        #: The exit code for this exception
        exit_code = 1

        def __init__(self, message):
            ctor_msg = message
            if PY2:
                if ctor_msg is not None:
                    ctor_msg = ctor_msg.encode('utf-8')
            Exception.__init__(self, ctor_msg)
            self.message = message

        def format_message(self):
            return self.message

        def __unicode__(self):
            return self.message

        def __str__(self):
            return self.message.encode('utf-8')

        def show(self, file=None):
            if file is None:
                file = get_text_stderr()
            echo('Error: %s' % self.format_message(), file=file)

In python 3 you pass the message as is (it might cause another issue, but about that later).

In python 2 you immediately encode the passed variable using utf-8. This means that you're expecting argument to be of unicode type, but at the same time you're discouraging users to use unicode_literals, and in most situations users will pass a regular string.

In python 3 __str__ is trying to convert text to bytes, while the message would already be text. This most of the times will look correct, but it might spew garbage when there are non ascii characters.

Here's corrected code (did not test it though):

    if PY2:
      unicode = str

    class ClickException(Exception):
        """An exception that Click can handle and show to the user."""

        #: The exit code for this exception
        exit_code = 1

        def __init__(self, message):
            if PY2 and isinstance(self, unicode):
                message = message.encode()
            Exception.__init__(self, message)
            self.message = message

        def format_message(self):
            return self.message

        def __unicode__(self):
            return str(self).decode()

        def __str__(self):
            return str(self.message)

        def show(self, file=None):
            if file is None:
                file = get_text_stderr()
            echo('Error: %s' % self.format_message(), file=file)

In python 3 for example click refuses to run if LANG and LC_CTYPE are not defined. Why doesn't it simply do what all other applications are doing and simply fall back to latin-1 (ISO-8859-1) instead of printing the error.

Also, another issue (and my above code is also is affected by this). Click is checking for environmental variables before continuing, yet encode and decode have hardcoded utf-8. It probably should use whatever locale.getdefaultlocale() returns.

For some locales Windows has additional bonus, the Ansi (GUI) encoding is different than OEM (console) and it was heroic undertaking to make your program work correctly for both.

Python3 works out of the box.

For true insanity try adding IDLE to the mix. IDLE under Windows in Python 2 will let you output UTF-8, but won't accept it as input. Python 3 is more consistent.

https://simonsapin.github.io/wtf-8/

Dunno about you, but the amount of encoding/decoding errors in Py2 broke my little code multiple times in the past. "oh yeah I'll just fetch this html page"... all good, until that page contains non-ASCII and then BOOM.

With Py3 you're forced to think about this sort of issues right away and with minimal fuss you're sorted for life.

And it's not like Python 3 is forcing you to give up bytes. You can still os.listdir(b'/some/dir') and you'll get a list of bytes objects back. It's just that that's not default, since using Unicode by default simply makes more sense on a cross-platform system.