Re:WorstFit: Unveiling Hidden Transformers in Windows ANSI
by Joker_vD on Friday January 10, 2025 @09:56AM
(#42656100)
Not everyone uses Latin-based scripts, you know. Most of the symbols in the BMP (including Brahmic scripts) take two bytes in either UTF-8 or UTF-16, and CJK symbols take 3 bytes in UTF-8 instead of 2 in UTF-16. Emojis, again, are 4 bytes long in either encoding. So for the most people in the world, UTF-16 is either slightly more compact encoding, or literally the same as UTF-8.
[ Parent
| on HN ]
Re:WorstFit: Unveiling Hidden Transformers in Windows ANSI
by account42 on Wednesday January 15, 2025 @08:32AM
(#42710539)
> Not everyone uses Latin-based scripts
Actually, everyne does use Latin-based scripts extensively. Maybe not exclusively but your almost all text-like data intended to be consumed by programs will mainly be Latin-based scripts. So even for languages where you have characters that need 3-bytes in UTF-8 but two in UTF-16 you can still end up saving memory with UTF-8 because all the boilerplate syntax around your fancy characters is ASCII.