Re:How UTF-8 Works
by morelisp on Tuesday February 08, 2022 @03:54PM
(#30264550)
Python 3 encoding management was broken, because it tried to impose Unicode semantics on things that were actually byte streams. For anyone actually correctly handling encodings in Python 2 it was awful because suddenly the language runtime was hiding half the data you needed.
[ Parent
| on HN ]
Re:How UTF-8 Works
by BiteCode_dev on Tuesday February 08, 2022 @05:29PM
(#30265765)
Nowadays, passing bytes to any os function returns bytes objects, not unicode. You'll get string if you pass string objects though, and they will be using utf8 surrogate escaping.
And nowadays, lots of people left the Python ecosystem completely because the 3 upgrade was broken and raising so many UnicodeDecodingErrors for so long. I'm glad it's fixed but it cost too much.
(And it had nothing to do with UTF-8. Actually, it's at least partially caused by the CPython developers avoiding UTF-8 for poor reasons.)
I'd argue that the people correctly handling encodings in Python 2 were vastly outnumbered by the people that weren't, but were getting away with it because the code didn't outright crash. Now in Python 3 it crashes, which is a pain in the short term but better in the long run.
I don't really think it's better - certainly it was not better until many years after release when they started admitting their big mistakes.
But even today, codepoint indexing is too easy and doesn't crash so lots of code is still subtly wrong. Memory usage of a string grows 3x if you add a single emoji. Most libraries are still agnostic about whether they take bytes or str so you still get exceptions thrown with no easy solution. (The growing popularity of type hints is fixing this, but that's not really related to Python 3.)