News for hackers. Stuff that matters.
Re:The Tragedy of UCS-2 by raphlinus on Saturday August 03, 2019 @10:30AM (#20600570)
I would go further and say that UCS-4 is also a variable length encoding, mostly because of emoji and its endearing ZWJ sequences, not to mention the encoding of flags as two Regional Indicator Symbol codepoints (both in the astral plane, of course), which makes Unicode not self-synchronizing.

So the idea of ever having a fixed-length encoding for Unicode is basically impossible now. Best to just use UTF-8 for everything and logic to group it in to code points, grapheme clusters, or whatever other granularity is needed.

Re:The Tragedy of UCS-2 by speleo_engr on Saturday August 03, 2019 @02:05PM (#20601745)
Those are all shaping (and bidi) issues though, independent from encoding. UCS-4 is a single Unicode code point and the simplest encoding. Shaping may combine code points many to one, one to many, have ordering issues, etc. I think it is important for people to understand how shaping works and the code point/glyph distinction. Trying to duplicate the effort of the shaper when processing strings UCS-4 is going to lead to a world of hurt. Even English text with strings like "1/4" and ligatures in the font may display as a single glyph (that's font specific shaping, not encoding).

I've heard more than one person tell me they don't need to worry about text shaping since they are using UTF-8. (That statement doesn't make any sense) There is a lot of confusion with Unicode text rendering stack.

Re:The Tragedy of UCS-2 by raphlinus on Saturday August 03, 2019 @02:30PM (#20601857)
Sure. I guess what I'm trying to say is: when do you care about code point boundaries at all? In my experience, it's actually fairly rare. If you're drawing text, you care about the result after shaping. If you're moving the cursor, you care about grapheme cluster boundaries. If you're doing stuff like sorting and collation, you should use specialized Unicode libraries that get this stuff right. When is the last time you wrote code that really needed to iterate Unicode code points specifically?
Re:The Tragedy of UCS-2 by cesarb on Saturday August 03, 2019 @02:53PM (#20601971)
> When is the last time you wrote code that really needed to iterate Unicode code points specifically?

We do it all the time. Splitting a string at U+002C or U+003B or U+000A, looking for U+007B or U+007D, etc. In my experience, it's really common.

Re:The Tragedy of UCS-2 by raphlinus on Saturday August 03, 2019 @02:57PM (#20601987)
Right, and if you're doing processing of ASCII code points (this is 95% of Markdown parsing, for example, the major exception being case folding and Unicode whitespace), then it's just as convenient to do those in UTF-8 as any other representation.
Re:The Tragedy of UCS-2 by a1369209993 on Saturday August 03, 2019 @06:27PM (#20603119)
Those aren't unicode code points; those are bytes.
Re:The Tragedy of UCS-2 by garmaine on Saturday August 03, 2019 @08:40PM (#20603726)
Not on a UTF-16 platform.
Re:The Tragedy of UCS-2 by Jasper_ on Saturday August 03, 2019 @10:39AM (#20600625)
Fun fact: because flags are a political nature, and Unicode did not want to limit the length of country codes, any number of Regional Indicator Symbol code points can be placed back to back and treated as an extended grapheme cluster.

That means that what Unicode today considers "one character" can be infinite in size.

Re:The Tragedy of UCS-2 by kps on Saturday August 03, 2019 @11:35AM (#20600893)
And worse, the ‘operators’ are postfix or infix, so if you're handling text sequentially — and let's say, split across network packets with an arbitrary delay in between — you can't even tell you've read a complete character until you receive the next one.

Typing in many Latin languages generates diacritics using ‘dead keys’, which are prefix operators. (Originally, on a physical typewriter, these were keys that simply struck the paper without triggering the mechanism to advance the carriage.) If Unicode had taken this hint, life would be easier.

(ISO/IEC 6937 had prefix diacritics, but it was too late.)

Re:The Tragedy of UCS-2 by tialaramex on Saturday August 03, 2019 @11:56AM (#20601014)
Complete recognition before processing. It's not the law, just a good idea, but you should do it anyway. In cryptography this lesson had to keep being learned. "Oh, I received this partial data, and I processed it, and then in the next packet I received the MAC and I realised the bad guys altered the data, oops"

Process whole strings, the meaning of a partial string may not be what you hope, even without fancy writing systems and Unicode encoding.

"Give the money to Steph" - OK, will do

"...enson" - Crap, OK, somebody chase Steph and get our money back, meanwhile here's James Stephenson's money

"... once he gives you the key" - Aargh. Somebody chase down James and get the key off him.

Re:The Tragedy of UCS-2 by lanstin on Saturday August 03, 2019 @12:27PM (#20601188)
Wow so interesting an idea. But so many streaming algorithms etc. people stick megabytes of crap into one framed message.
Re:The Tragedy of UCS-2 by Dylan16807 on Saturday August 03, 2019 @01:08PM (#20601436)
Sure, but you can also just stack infinite accent marks.

Either way there's already a recommendation in this area. I'm not sure if it strictly fits this particular scenario but the "UAX15-D3. Stream-Safe Text Format" suggests that you should support sequences at least 31 code points long.

Re:The Tragedy of UCS-2 by Latty on Saturday August 03, 2019 @01:27PM (#20601545)
To be fair, I believe this was true well before flags—diacritics played this role previously, as in "Zalgo Text".
Re:The Tragedy of UCS-2 by wbl on Saturday August 03, 2019 @11:38AM (#20600913)
What do you think a character is? ZWJ and combining characters are needed for a few Indian languages spoken by millions of people, more then speak Dutch or Norwegian.
Re:The Tragedy of UCS-2 by khaled on Saturday August 03, 2019 @04:47PM (#20602577)
Some low level text libraries (FriBiDi, HarfBuzz) use UTF-32 as it makes code simpler when you have to work on it at code point level. Most code does not need that of course and is better off working as a different level than code points.
Re:The Tragedy of UCS-2 by a1369209993 on Saturday August 03, 2019 @06:23PM (#20603097)
> both in the astral plane, of course

Well never mind that then; how do you like U+41,U+300 aka U+C0 latin capital letter a with grave accent. Not just in the BMP but in the part of the BMP that's only 2 bytes of UTF-8.

And there's a whole bunch of similar characters that don't have NFC encodings.