News for hackers. Stuff that matters.
Re:How UTF-8 Works by chrismorgan on Tuesday February 08, 2022 @10:54PM (#30268592)
The really tragic thing is that UTF-8 was invented before UTF-16. But a few big companies had put a couple of years of heavy investment into UCS-2, and weren’t willing to let that and the attendant pain they were beginning to foist on developers and users alike go to waste, and so ruined Unicode with UTF-16 and the disaster called surrogates that is the cause of the significant majority of programming languages handling strings incorrectly (e.g. JavaScript uses potentially ill-formed Unicode, indexed by UTF-16 code unit; and Python strings are sequences of code points rather than scalar values). If only they had reversed course and said “sorry for all the pain we were just starting to put you through, that fixed-width 16-bit encoding thing didn’t pan out, we’re going back to 8-bit encodings with this UTF-8 thing that is conveniently also backwards-compatible with ASCII”. If only.
Re:How UTF-8 Works by account42 on Wednesday February 09, 2022 @06:35AM (#30271502)
> the disaster called surrogates that is the cause of the significant majority of programming languages handling strings incorrectly

It also wastes valuable code space that could be used to have more characters encode as 3 bytes instead of 4 in UTF-8.