Simple Dynamic Strings library for C, compatible with null-terminated strings

Re:Simple Dynamic Strings library for C, compatible with nul... by asveikau on Tuesday December 03, 2019 @12:23PM (#21694367)

I don't get entirely why people want or expect to store UTF-32 in memory for any substantial period. It makes more sense to me as: if you need to process codepoint at a time, parse from UTF-8 or UTF-16 in a codepoint-at-a-time fashion, leaving only ~1 codepoint decoded at a time.

People seem to think that 1 codepoint = 1 integer frees you from thinking Unicode is hard. But 1 codepoint is not 1 glyph. You have combining characters, zero-width joiners [used also in emojis], RTL markers, Han unification, probably more. So you can't really think of a Unicode string as a random-access, one-glyph-per-unit type of thing in any encoding.

So I hope when people say "UTF-32 support" they mean "decode UTF-8".

[ Parent | on HN ]

Re:Simple Dynamic Strings library for C, compatible with nul... by zzo38computer on Wednesday December 04, 2019 @12:37AM (#21699883)

The Unicode functions of Glk use UTF-32 (and numbers outside of Unicode range are used for function keys), and Glulx supports strings of 32-bit characters (usually Unicode, although this is not required) (strings of 8-bit characters are also supported, but this doesn't support UTF-8 or any other multibyte encoding). (However, strings in Glulx are usually huffed anyways, so other than the temporary buffers you would ordinarily store a variable number of bits per character (or sometimes a string of several characters has its own code), and can be shorter than using UTF-8 or UTF-32 or other uncompressed encodings.)

The stuff you mention does make Unicode very messy though.

[ Parent | on HN ]