Re:Simple Dynamic Strings library for C, compatible with nul...
by Keyframe on Tuesday December 03, 2019 @11:51AM
(#21694011)
Unfortunately, yes. Limited usefulness at best without unicode support, at least to a degree. Even UTF-16 or 32 internally would suffice, treating UTF-8 only as ser/de format is good enough these days.
[ Parent
| on HN ]
Re:Simple Dynamic Strings library for C, compatible with nul...
by asveikau on Tuesday December 03, 2019 @12:23PM
(#21694367)
I don't get entirely why people want or expect to store UTF-32 in memory for any substantial period. It makes more sense to me as: if you need to process codepoint at a time, parse from UTF-8 or UTF-16 in a codepoint-at-a-time fashion, leaving only ~1 codepoint decoded at a time.
People seem to think that 1 codepoint = 1 integer frees you from thinking Unicode is hard. But 1 codepoint is not 1 glyph. You have combining characters, zero-width joiners [used also in emojis], RTL markers, Han unification, probably more. So you can't really think of a Unicode string as a random-access, one-glyph-per-unit type of thing in any encoding.
So I hope when people say "UTF-32 support" they mean "decode UTF-8".
Re:Simple Dynamic Strings library for C, compatible with nul...
by zzo38computer on Wednesday December 04, 2019 @12:37AM
(#21699883)
The Unicode functions of Glk use UTF-32 (and numbers outside of Unicode range are used for function keys), and Glulx supports strings of 32-bit characters (usually Unicode, although this is not required) (strings of 8-bit characters are also supported, but this doesn't support UTF-8 or any other multibyte encoding). (However, strings in Glulx are usually huffed anyways, so other than the temporary buffers you would ordinarily store a variable number of bits per character (or sometimes a string of several characters has its own code), and can be shorter than using UTF-8 or UTF-32 or other uncompressed encodings.)
The stuff you mention does make Unicode very messy though.
Re:Simple Dynamic Strings library for C, compatible with nul...
by jstimpfle on Tuesday December 03, 2019 @12:18PM
(#21694303)
UTF-8 is the preferred internal storage format for most applications. The reason is space efficiency.
Re:Simple Dynamic Strings library for C, compatible with nul...
by jandrese on Tuesday December 03, 2019 @01:20PM
(#21694991)
That and the perceived simplicity of UTF16/32 is not actually the case so why bother if you have to do it the hard way anyway.
Re:Simple Dynamic Strings library for C, compatible with nul...
by cardiffspaceman on Tuesday December 03, 2019 @01:43PM
(#21695234)
I certainly have leaned on UTF-8 but I wonder how efficient it is for the numerically-higher code points if the language is not heavily cp1252, like Korean.
Re:Simple Dynamic Strings library for C, compatible with nul...
by pstch on Wednesday December 04, 2019 @12:27AM
(#21699845)
An interesting - but not surprising - thing about this is that compression algorithms can be more efficient on wider representations of numerically-high code points (e.g, for some Korean corpus, using UTF-32 instead of UTF-8 improves LZMA compression by ~10%).
Re:Simple Dynamic Strings library for C, compatible with nul...
by zzo38computer on Wednesday December 04, 2019 @12:39AM
(#21699895)
How well does that corpus compress with LZMA if using a Korean specific character code (such as EUC-KR)? And what about other combinations, with other character codings and other compression algorithms?
Re:Simple Dynamic Strings library for C, compatible with nul...
by pstch on Wednesday December 04, 2019 @01:52AM
(#21700166)
EUC-KR doesn't improve much with LZMA (2% over UTF-16), but is better with gzip-9 (10% over UTF-16). I haven't studied this extensively, just did a few tests when waiting for it to download.