Re:Plain Text Doesn’t Exist: Unicode and encodings demystified
by stormbrew on Monday August 26, 2013 @08:41PM
(#6280782)
> However, programming languages that use UTF-8 as the primary string representation tend to have inferior Unicode support to those that use UTF-16.
I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.
Re:Plain Text Doesn’t Exist: Unicode and encodings demystified
by plorkyeran on Monday August 26, 2013 @12:14PM
(#6277951)
UTF-16 has the same issue with invalid things appearing to work due to surrogate pairs. A lot of UTF-16 based applications actually only support UCS-2, because you can assume one string index equals one character and get away with it if it never occurs to you to try to use stuff outside of the BMP.
For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.