News for hackers. Stuff that matters.
Re:Simple Dynamic Strings library for C, compatible with nul... by jstimpfle on Tuesday December 03, 2019 @12:18PM (#21694303)
UTF-8 is the preferred internal storage format for most applications. The reason is space efficiency.
Re:Simple Dynamic Strings library for C, compatible with nul... by jandrese on Tuesday December 03, 2019 @01:20PM (#21694991)
That and the perceived simplicity of UTF16/32 is not actually the case so why bother if you have to do it the hard way anyway.
Re:Simple Dynamic Strings library for C, compatible with nul... by cardiffspaceman on Tuesday December 03, 2019 @01:43PM (#21695234)
I certainly have leaned on UTF-8 but I wonder how efficient it is for the numerically-higher code points if the language is not heavily cp1252, like Korean.
Re:Simple Dynamic Strings library for C, compatible with nul... by pstch on Wednesday December 04, 2019 @12:27AM (#21699845)
An interesting - but not surprising - thing about this is that compression algorithms can be more efficient on wider representations of numerically-high code points (e.g, for some Korean corpus, using UTF-32 instead of UTF-8 improves LZMA compression by ~10%).
Re:Simple Dynamic Strings library for C, compatible with nul... by zzo38computer on Wednesday December 04, 2019 @12:39AM (#21699895)
How well does that corpus compress with LZMA if using a Korean specific character code (such as EUC-KR)? And what about other combinations, with other character codings and other compression algorithms?
Re:Simple Dynamic Strings library for C, compatible with nul... by pstch on Wednesday December 04, 2019 @01:52AM (#21700166)
EUC-KR doesn't improve much with LZMA (2% over UTF-16), but is better with gzip-9 (10% over UTF-16). I haven't studied this extensively, just did a few tests when waiting for it to download.