News for hackers. Stuff that matters.
Re:Why to normalize Unicode strings by ameliaquining on Wednesday March 13, 2019 @06:14PM (#19384466)
Also "legacy" language runtimes. "Legacy" being in scare quotes because JavaScript, the JVM, and the CLR all work this way and are all very much in widespread use today.
Re:Why to normalize Unicode strings by Someone on Wednesday March 13, 2019 @07:39PM (#19385112)
JDK 9 introduced “compact strings” (https://bugs.openjdk.java.net/browse/JDK-8054307). That stores a string’s characters in a byte array, with either the traditional two bytes per ‘char’ (encoding the entire string as UTF16) or, if possible, one (encoding the entire string as ISO-8859-1/Latin-1). They probably didn’t use UTF-8 because it would break the fact that string indexing is O(1).
Re:Why to normalize Unicode strings by int_19h on Wednesday March 13, 2019 @09:38PM (#19385796)
That's just internal representation, though. Semantically, strings are still sequences of chars, and char is still 16-bit, so the API is still UTF-16.
Re:Why to normalize Unicode strings by ken on Wednesday March 13, 2019 @07:17PM (#19384963)
Yes, indeed. I didn't mean "legacy" to imply that they aren't still useful. Simply that many of them predate the creation of codepoints which don't fit in 16 bits -- and it's harder to upgrade a binary API distributed in a client OS, compared to changing a web server's preferred encoding.

JS/JVM/CLR all work fine, but I imagine if they were created today, their strings would not be based on UTF-16.

Re:Why to normalize Unicode strings by sebazzz on Thursday March 14, 2019 @02:20AM (#19386884)
Utf8String is in the works for C#/.NET. It would still be pretty much bolted on though.