News for hackers. Stuff that matters.
Re:Why to normalize Unicode strings by ken on Wednesday March 13, 2019 @05:23PM (#19383994)
> Why use both [UTF-8 and UTF-16]? Western languages typically are most efficiently encoded with UTF-8 (since most characters would be represented with 1 byte only), while Asian languages can usually produce smaller files when using UTF-16 as encoding.

The second sentence is technically correct, but it's a strange followup here because it's not why UTF-8 and UTF-16 exist today. I don't know any Asian webpages that use UTF-16 to save bandwidth, e.g., Japanese Wikipedia is still UTF-8.

The major use of UTF-16 in 2019, AFAICT, is for legacy operating system interfaces.

Re:Why to normalize Unicode strings by mjevans on Wednesday March 13, 2019 @06:17PM (#19384483)
Well, the second sentence isn't often correct.

It'd probably be true for pure TEXT files IIRC, however the markup elements of webpages and javascript also take up space and /those/ are western elements. On actual web pages UTF-8 is actually the smallest choice.

Re:Why to normalize Unicode strings by ameliaquining on Wednesday March 13, 2019 @06:14PM (#19384466)
Also "legacy" language runtimes. "Legacy" being in scare quotes because JavaScript, the JVM, and the CLR all work this way and are all very much in widespread use today.
Re:Why to normalize Unicode strings by Someone on Wednesday March 13, 2019 @07:39PM (#19385112)
JDK 9 introduced “compact strings” (https://bugs.openjdk.java.net/browse/JDK-8054307). That stores a string’s characters in a byte array, with either the traditional two bytes per ‘char’ (encoding the entire string as UTF16) or, if possible, one (encoding the entire string as ISO-8859-1/Latin-1). They probably didn’t use UTF-8 because it would break the fact that string indexing is O(1).
Re:Why to normalize Unicode strings by int_19h on Wednesday March 13, 2019 @09:38PM (#19385796)
That's just internal representation, though. Semantically, strings are still sequences of chars, and char is still 16-bit, so the API is still UTF-16.
Re:Why to normalize Unicode strings by ken on Wednesday March 13, 2019 @07:17PM (#19384963)
Yes, indeed. I didn't mean "legacy" to imply that they aren't still useful. Simply that many of them predate the creation of codepoints which don't fit in 16 bits -- and it's harder to upgrade a binary API distributed in a client OS, compared to changing a web server's preferred encoding.

JS/JVM/CLR all work fine, but I imagine if they were created today, their strings would not be based on UTF-16.

Re:Why to normalize Unicode strings by sebazzz on Thursday March 14, 2019 @02:20AM (#19386884)
Utf8String is in the works for C#/.NET. It would still be pretty much bolted on though.
Re:Why to normalize Unicode strings by profmonocle on Wednesday March 13, 2019 @07:13PM (#19384937)
I imagine compression negates most of the overhead from encoding Asian web pages in UTF-8. Plus, considering how image/markup/script heavy modern web pages are, the actual human-readable text is a tiny sliver of most sites' bandwidth costs.