Re:Crafting Interpreters: Strings
by ufo on Monday September 24, 2018 @06:48PM
(#18062155)
However, this is only the case if you have just a big blob of text and UTF-8 might be better if the east asian text is interspersed with ASCII characters. For example, HTML with east asian text often is smaller when encoded with UTF-8, because a significant portion of the content is HTML markup.
[ Parent
| on HN ]
Right well again, if your average East Asian codepoint is 3 bytes in UTF-8, and your average markup codepoint is one byte, then as the percentage of markup in your corpus rises you'll grow asymptotically towards 1. But not all text contains markup ;) Consider any database storage for example, text in a game, e-books, visual novels. I guess anything that isn't the web or JSON is what I'm saying -- which is a lot.
Re:Crafting Interpreters: Strings
by a1369209993 on Tuesday September 25, 2018 @02:12AM
(#18064003)
Also bear in mind that gzip (or any non-toy compression algorithm) will do much better than either on any non-contrived input.