Re:Why to normalize Unicode strings
by ameliaquining on Wednesday March 13, 2019 @06:14PM
(#19384466)
Also "legacy" language runtimes. "Legacy" being in scare quotes because JavaScript, the JVM, and the CLR all work this way and are all very much in widespread use today.
[ Parent
| on HN ]
JDK 9 introduced “compact strings” (https://bugs.openjdk.java.net/browse/JDK-8054307). That stores a string’s characters in a byte array, with either the traditional two bytes per ‘char’ (encoding the entire string as UTF16) or, if possible, one (encoding the entire string as ISO-8859-1/Latin-1). They probably didn’t use UTF-8 because it would break the fact that string indexing is O(1).
That's just internal representation, though. Semantically, strings are still sequences of chars, and char is still 16-bit, so the API is still UTF-16.
Yes, indeed. I didn't mean "legacy" to imply that they aren't still useful. Simply that many of them predate the creation of codepoints which don't fit in 16 bits -- and it's harder to upgrade a binary API distributed in a client OS, compared to changing a web server's preferred encoding.
JS/JVM/CLR all work fine, but I imagine if they were created today, their strings would not be based on UTF-16.