The sad history of Unicode printf-style format specifiers in Visual C++

Re:The sad history of Unicode printf-style format specifiers... by Mikhail_Edoshin on Thursday December 01, 2022 @07:09AM (#33815211)

This is plain decoding of a trusted string:

    utf16 decode
      if (unit <= 0xD7FF || unit >= 0xE000)
        /* one unit */;
      else
        /* two units */
    utf-8 decode
      if (unit < 0xE0) {
        if (unit < 0xC0)
          /* one unit */
        else
          /* two units */
      }
      else if (unit < 0xF0)
        /* three units */
      else
        /* four units */

UTF-16 will take one comparison for all characters up to 0xD7FF; UTF-8 normally takes two. Same when encoding:

    utf16 encode
      if (value <= 0xFFFF)
        /* one unit */
      else
        /* two units */
    utf8 encode
      if (value <= 0x7FF) {
        if (value <= 0x7F)
          /* one unit */
        else
          /* two units */
      }
      else if (value <= 0xFFFF)
        /* three units */
      else
        /* four units */

(We can tune UTF-8 it to take one comparison for ASCII if we expect mostly ASCII.)

Things get even more complex when we are to read an untrusted string. In UTF-8 we have five byte types, invalid bytes, conditionally invalid bytes, and must recognize surrogates (as errors). In UTF-16 all 16-bit units are valid and there are only three unit types: character or surrogate, high or low. I once wrote a decoder for UTF-8/16 and here's my stats:

    UTF-8    : 13 byte types, 13 states, 10 actions
    UTF-16LE :  3 byte types,  4 states,  5 actions

[ Parent | on HN ]

Re:The sad history of Unicode printf-style format specifiers... by SaddledBounding on Thursday December 01, 2022 @10:30AM (#33817546)

utf-8 has an elegance that utf-16 lacks:

    utf-8 decode
      switch (std::countl_one(unit)) {
        case 0:
          /* one unit */
          break;
        case 2:
          /* two units */
          break;
        case 3:
          /* three units */
          break;
        case 4:
          /* four units */
          break;
        default:
          /* not code point boundary */
          break;
      }

[ Parent | on HN ]

Re:The sad history of Unicode printf-style format specifiers... by Mikhail_Edoshin on Friday December 02, 2022 @12:01AM (#33826992)

Oh, thanks, this one is good. Must be rather fast on major platforms. But with this optimization UTF-8 still remains harder computationally than UTF-16.

[ Parent | on HN ]

Re:The sad history of Unicode printf-style format specifiers... by bergesenha on Friday December 02, 2022 @02:18AM (#33827807)

I’m happy every time I see c++20 in the wild

[ Parent | on HN ]