News for hackers. Stuff that matters.
Re:The sad history of Unicode printf-style format specifiers... by heinrich5991 on Thursday December 01, 2022 @05:47AM (#33814646)
Can you explain in what way it is easier to handle than UTF-8? Most code that I've seen and looks easier just mishandles characters outside the basic multilingual plane. Thankfully, due to emojis being a thing, a lot of developers woke up to the fact that Unicode specifies more than 65536 characters.
Re:The sad history of Unicode printf-style format specifiers... by Const-me on Thursday December 01, 2022 @06:33AM (#33814977)
UTF-16 is cheaper to decode. All CPUs have branch predictors, 99.99% of all UTF-16 strings don't contain any surrogate pairs.

For East Asian languages UTF-16 uses 50% less memory, these characters take 3 bytes in UTF-8, but only 2 bytes in UTF-16.

Re:The sad history of Unicode printf-style format specifiers... by barrkel on Thursday December 01, 2022 @09:27AM (#33816623)
That's 33% less memory, or conversely, UTF-8 uses 50% more memory.
Re:The sad history of Unicode printf-style format specifiers... by Mikhail_Edoshin on Thursday December 01, 2022 @07:09AM (#33815211)
This is plain decoding of a trusted string:

    utf16 decode
      if (unit <= 0xD7FF || unit >= 0xE000)
        /* one unit */;
      else
        /* two units */
    utf-8 decode
      if (unit < 0xE0) {
        if (unit < 0xC0)
          /* one unit */
        else
          /* two units */
      }
      else if (unit < 0xF0)
        /* three units */
      else
        /* four units */
UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding:

    utf16 encode
      if (value <= 0xFFFF)
        /* one unit */
      else
        /* two units */
    utf8 encode
      if (value <= 0x7FF) {
        if (value <= 0x7F)
          /* one unit */
        else
          /* two units */
      }
      else if (value <= 0xFFFF)
        /* three units */
      else
        /* four units */
(We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.)

Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats:

    UTF-8    : 13 byte types, 13 states, 10 actions
    UTF-16LE :  3 byte types,  4 states,  5 actions
Re:The sad history of Unicode printf-style format specifiers... by SaddledBounding on Thursday December 01, 2022 @10:30AM (#33817546)
utf-8 has an elegance that utf-16 lacks:

    utf-8 decode
      switch (std::countl_one(unit)) {
        case 0:
          /* one unit */
          break;
        case 2:
          /* two units */
          break;
        case 3:
          /* three units */
          break;
        case 4:
          /* four units */
          break;
        default:
          /* not code point boundary */
          break;
      }
Re:The sad history of Unicode printf-style format specifiers... by Mikhail_Edoshin on Friday December 02, 2022 @12:01AM (#33826992)
Oh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.
Re:The sad history of Unicode printf-style format specifiers... by bergesenha on Friday December 02, 2022 @02:18AM (#33827807)
I’m happy every time I see c++20 in the wild