Moving Away from Python 2

Re:Moving Away from Python 2 by takeda on Saturday May 21, 2016 @12:13AM (#11743173)

> Which works great up until the point that some existing data that's supposed to be text, like a filename, or a file in a text-based format, or string fields in a file in a binary format, et cetera, turns out to be invalid UTF-8. Maybe because it was written in the wrong encoding, or because some binary data has been stuffed in (e.g. with heredoc quoting), or due to corruption, or even maliciously as an obfuscation measure. And Py3 turns up its nose and rejects it, probably by throwing an exception and killing the script - while programs written in other languages, including Py2 (if the unicode type is avoided), 'just work'.

I'm tired of people spreading FUD about it. they either don't understand what they are doing (and do it incorrectly) or are repeating what other people (who did things incorrectly) said:

    » echo $LANG
    en_US.UTF-8
    » echo $LC_CTYPE
    en_US.UTF-8
    » echo 'works' > $'\377'$'\377'$'\377'
    » ls
    ???
    » cat $'\377'$'\377'$'\377'
    works
    » python3
    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> a = b'\377\377\377'
    >>> a
    b'\xff\xff\xff'
    >>> a.decode()
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
    >>> b = open(a)
    >>> b.read()
    'works\n'
    >>>

Accessing files with invalid utf-8 characters works just fine. Being explicit and clearly distinguishing between a string and bytes allows python to know when the filename is encoded using utf-8 (or in fact any other encoding you have defined in your environment) or bytes. Actually python2 struggles here, because it treats everything as bytes and can't tell what is used. Yes, legacy python probably won't crash, but it will spew garbage.

Python3 on the other side is strict about to point errors in your code. If python3 code will crash it's either bug in python or (most likely) bug in your code and you should fix it.

I (and I'm sure a lot of people) prefer program to crash, pointing where the error is, than silently corrupt data once in a while, and when you notice the issue spending days or months figuring out where the corruption is happening.

[ Parent | on HN ]

Re:Moving Away from Python 2 by comex on Saturday May 21, 2016 @02:19AM (#11743448)

Okay, but where would that filename come from in a real script? If it was passed in argv, then apparently Py3 gives you a string which is invalid Unicode (contains unpaired surrogates), and explodes if you try to print it or do anything else that implicitly encodes it without the special 'surrogateescape' setting. It's really dangerous to pass such a string around. But at least you can open it, so it's not completely broken. Oh, and that means that two different strings can encode to the same path, which... is probably not usually a problem...

On the other hand, if you read a list of filenames from a file, you'd better remember to either use binary or explicitly choose surrogateescape decoding. Which is a special case of "file in a text-based format" from my original post.

I don't know what you mean by "corruption".

[ Parent | on HN ]

Re:Moving Away from Python 2 by takeda on Tuesday May 24, 2016 @01:27AM (#11759156)

What I meant by corruption is the program might be spewing garbage, instead of simply crashing pointing where encoding problem is.

As for your other point. You're right, but there's no easy way resolving this. Either treat everything as unicode and remember it's binary on linux, or have linux working and everything else broken.

In that scenario you probably should use .encode('utf-8', 'surrogateescape') as soon as you can and work on bytes, so no need of doing the above.

Also, if you for example call os.listdir() with argument in bytes it'll return entries in bytes:

    Python 3.4.1 (default, May 23 2014, 17:48:28) [GCC] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import os
    >>> os.listdir(b'.')
    [b'\xff\xff\xff', b'test.py']

But IMO, things still could be better. I would love if there was matching sys.setfiliesystemencoding() to sys.getfilesystemencoding(). In scenarios when you are not sure of encoding you could fall back to iso-8859-1 (latin-1) which has 1:1 mapping for bytes.

Currently looks like you would have to use:

    filename.encode(sys.getfilesystemencoding(), 'surrogateescape').decode('iso-8859-1')

[ Parent | on HN ]