News for hackers. Stuff that matters.
Re:The Tragedy of UCS-2 by speleo_engr on Saturday August 03, 2019 @02:05PM (#20601745)
Those are all shaping (and bidi) issues though, independent from encoding. UCS-4 is a single Unicode code point and the simplest encoding. Shaping may combine code points many to one, one to many, have ordering issues, etc. I think it is important for people to understand how shaping works and the code point/glyph distinction. Trying to duplicate the effort of the shaper when processing strings UCS-4 is going to lead to a world of hurt. Even English text with strings like "1/4" and ligatures in the font may display as a single glyph (that's font specific shaping, not encoding).

I've heard more than one person tell me they don't need to worry about text shaping since they are using UTF-8. (That statement doesn't make any sense) There is a lot of confusion with Unicode text rendering stack.

Re:The Tragedy of UCS-2 by raphlinus on Saturday August 03, 2019 @02:30PM (#20601857)
Sure. I guess what I'm trying to say is: when do you care about code point boundaries at all? In my experience, it's actually fairly rare. If you're drawing text, you care about the result after shaping. If you're moving the cursor, you care about grapheme cluster boundaries. If you're doing stuff like sorting and collation, you should use specialized Unicode libraries that get this stuff right. When is the last time you wrote code that really needed to iterate Unicode code points specifically?
Re:The Tragedy of UCS-2 by cesarb on Saturday August 03, 2019 @02:53PM (#20601971)
> When is the last time you wrote code that really needed to iterate Unicode code points specifically?

We do it all the time. Splitting a string at U+002C or U+003B or U+000A, looking for U+007B or U+007D, etc. In my experience, it's really common.

Re:The Tragedy of UCS-2 by raphlinus on Saturday August 03, 2019 @02:57PM (#20601987)
Right, and if you're doing processing of ASCII code points (this is 95% of Markdown parsing, for example, the major exception being case folding and Unicode whitespace), then it's just as convenient to do those in UTF-8 as any other representation.
Re:The Tragedy of UCS-2 by a1369209993 on Saturday August 03, 2019 @06:27PM (#20603119)
Those aren't unicode code points; those are bytes.
Re:The Tragedy of UCS-2 by garmaine on Saturday August 03, 2019 @08:40PM (#20603726)
Not on a UTF-16 platform.