Re:The sad history of Unicode printf-style format specifiers...
by Mikhail_Edoshin on Thursday December 01, 2022 @05:30AM
(#33814523)
No, I'm fully aware of it being a variable-length encoding. Yet as such it is still simpler to work with than UTF-8. So if we are to use a variable-length encoding anyway then UTF-16 is a reasonable choice. It is not compatible with ASCII and old APIs, that's true, but I cannot judge how important this is. I guess that for some projects the efficiency of UTF-16 should outweigh the inconveniences of that incompatibility.
[ Parent
| on HN ]
Re:The sad history of Unicode printf-style format specifiers...
by heinrich5991 on Thursday December 01, 2022 @05:47AM
(#33814646)
Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
Re:The sad history of Unicode printf-style format specifiers...
by Const-me on Thursday December 01, 2022 @06:33AM
(#33814977)
UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs.
For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.
Re:The sad history of Unicode printf-style format specifiers...
by barrkel on Thursday December 01, 2022 @09:27AM
(#33816623)
That's 33% less memory, or conversely, UTF-8 uses 50% more memory.
Re:The sad history of Unicode printf-style format specifiers...
by Mikhail_Edoshin on Thursday December 01, 2022 @07:09AM
(#33815211)
This is plain decoding of a trusted string:
utf16 decode
if (unit <= 0xD7FF || unit >= 0xE000)
/* one unit */;
else
/* two units */
utf-8 decode
if (unit < 0xE0) {
if (unit < 0xC0)
/* one unit */
else
/* two units */
}
else if (unit < 0xF0)
/* three units */
else
/* four units */
UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding: utf16 encode
if (value <= 0xFFFF)
/* one unit */
else
/* two units */
utf8 encode
if (value <= 0x7FF) {
if (value <= 0x7F)
/* one unit */
else
/* two units */
}
else if (value <= 0xFFFF)
/* three units */
else
/* four units */
(We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.)Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats:
UTF-8 : 13 byte types, 13 states, 10 actions
UTF-16LE : 3 byte types, 4 states, 5 actions
Re:The sad history of Unicode printf-style format specifiers...
by SaddledBounding on Thursday December 01, 2022 @10:30AM
(#33817546)
utf-8 has an elegance that utf-16 lacks:
utf-8 decode
switch (std::countl_one(unit)) {
case 0:
/* one unit */
break;
case 2:
/* two units */
break;
case 3:
/* three units */
break;
case 4:
/* four units */
break;
default:
/* not code point boundary */
break;
}
Re:The sad history of Unicode printf-style format specifiers...
by Mikhail_Edoshin on Friday December 02, 2022 @12:01AM
(#33826992)
Oh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.
Re:The sad history of Unicode printf-style format specifiers...
by bergesenha on Friday December 02, 2022 @02:18AM
(#33827807)
Iām happy every time I see c++20 in the wild
Re:The sad history of Unicode printf-style format specifiers...
by bigDinosaur on Thursday December 01, 2022 @06:32AM
(#33814971)
Can you show some concrete instances of it being easier to work with compared to UTF-8? As far as I can see you gain variable length encoding (albeit in a potentially confusing way) while losing ASCII support and also not gaining the pure simplicity of UTF-32. Oh and also endianness, although personally that's probably never an issue I will run into.
Re:The sad history of Unicode printf-style format specifiers...
by Mikhail_Edoshin on Thursday December 01, 2022 @07:33AM
(#33815393)
(Please see my reply to a sibling comment.)
Re:The sad history of Unicode printf-style format specifiers...
by bigDinosaur on Thursday December 01, 2022 @07:48AM
(#33815556)
Perfect, thanks for the heads up!