Well, the second sentence isn't often correct.
It'd probably be true for pure TEXT files IIRC, however the markup elements of webpages and javascript also take up space and /those/ are western elements. On actual web pages UTF-8 is actually the smallest choice.
Re:Why to normalize Unicode strings
by ameliaquining on Wednesday March 13, 2019 @06:14PM
(#19384466)
Also "legacy" language runtimes. "Legacy" being in scare quotes because JavaScript, the JVM, and the CLR all work this way and are all very much in widespread use today.
JDK 9 introduced “compact strings” (https://bugs.openjdk.java.net/browse/JDK-8054307). That stores a string’s characters in a byte array, with either the traditional two bytes per ‘char’ (encoding the entire string as UTF16) or, if possible, one (encoding the entire string as ISO-8859-1/Latin-1). They probably didn’t use UTF-8 because it would break the fact that string indexing is O(1).
That's just internal representation, though. Semantically, strings are still sequences of chars, and char is still 16-bit, so the API is still UTF-16.
Yes, indeed. I didn't mean "legacy" to imply that they aren't still useful. Simply that many of them predate the creation of codepoints which don't fit in 16 bits -- and it's harder to upgrade a binary API distributed in a client OS, compared to changing a web server's preferred encoding.
JS/JVM/CLR all work fine, but I imagine if they were created today, their strings would not be based on UTF-16.
Utf8String is in the works for C#/.NET. It would still be pretty much bolted on though.
Re:Why to normalize Unicode strings
by profmonocle on Wednesday March 13, 2019 @07:13PM
(#19384937)
I imagine compression negates most of the overhead from encoding Asian web pages in UTF-8. Plus, considering how image/markup/script heavy modern web pages are, the actual human-readable text is a tiny sliver of most sites' bandwidth costs.
The second sentence is technically correct, but it's a strange followup here because it's not why UTF-8 and UTF-16 exist today. I don't know any Asian webpages that use UTF-16 to save bandwidth, e.g., Japanese Wikipedia is still UTF-8.
The major use of UTF-16 in 2019, AFAICT, is for legacy operating system interfaces.