> In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character.
Only if people writing code are entirely oblivious to what UTF-8 is. The most basic feature of UTF-8 is that it's a variable-length encoding scheme. If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?
Re:Moving Away from Python 2
by ubernostrum on Saturday May 21, 2016 @10:04AM
(#11744699)
If the main feature is it's variable-length, why would any programmer assume the length doesn't vary?
Because UTF-8 is the Unicode encoding that looks just like ASCII, so long as the characters stay within the set supported by ASCII, so it lets them say they're doing "Unicode" while really just writing code to handle ASCII.
I am a Dane - we have three letters in our alphabet that are not represented in ascii (æ, ø and å) so let me assure you I don't except text to be in ascii, but I do expect it to be in UTF-8 and will consider it a bug if it isn't.
This is incorrect for anything other than an ASCII-only world. In general, "just use UTF-8" is a shorthand for "just let me pretend non-ASCII doesn't exist", because it leads to people writing code that assumes one byte == one character.
By rubbing your nose in the difference between str and bytes, Python 3 complicates your life.
Having had a lot of experience with it, I would instead say Python 3 makes you think about things and find bugs at the appropriate time -- up-front during the development process -- rather than at an inappropriate time, like 3AM on a Sunday when your pager goes off because you trusted the "helpful" but wrong way of doing things.