News for hackers. Stuff that matters.
Re:You can't just assume UTF-8 by gary_0 on Monday April 29, 2024 @07:31PM (#40205456)
For Asian languages, UTF-8 is basically the same size as any other encoding when compressed[0] (and you should be using compression if you care about space) so in practice there is no data size advantage to using non-standard encodings.

[0] https://utf8everywhere.org/#asian

Re:You can't just assume UTF-8 by neild on Monday April 29, 2024 @09:36PM (#40206363)
In addition, Chinese characters encode more information than English letters, so a text written in Chinese will generally consume fewer bytes than the same text in English even when using UTF-8.

(Consider: Horse is five letters, but 馬 is one character. Even at three bytes per character, Chinese wins.)

Re:You can't just assume UTF-8 by kstrauser on Monday April 29, 2024 @10:14PM (#40206570)
True, but even then you wouldn’t want to store it egregiously badly.
Re:You can't just assume UTF-8 by Panzer04 on Tuesday April 30, 2024 @01:17AM (#40207474)
Presumably that derives from the overhead of encoding an english character as a full byte? Given there's only 26 characters normally, you could fit that into 5 bits instead, which funnily enough does actually line up with the chinese character encoding (5x5 vs 1x24).
Re:You can't just assume UTF-8 by kps on Tuesday April 30, 2024 @11:03AM (#40211767)
Yes. It's the non-Latin alphabets that lose with either UTF-8 or UTF-16, compared with stateful ISO 2022 page switching.