For example, consider a function to open a file by name. If your strings are UTF-8, you can just pass your null terminated buffer to fopen() or something and things will work fine for most files. But if your strings are internally UTF-16, you have to think about which encoding to use, and you research it, and you discover that holy crap, this stuff differs across OSes, and so we better take this problem seriously.
I'm really curious which languages drove you to this conclusion. Right now we're in a state where the most mature implementations of unicode in general are utf-16 because of the ucs2 accident, so it wouldn't exactly surprise me, but I'd still like to see the proof.
But it's not like utf-16 was invented to solve that problem so much as it was an accident that in the early days of unicode 2 bytes was enough to encode all code points and a 2 byte encoding was appealing as a universal solution in a way that a 3 or 4 byte encoding never would be. Some group of people on the planet will have to suffer 3 or more byte encodings no matter what. And I'd be happy to make it mine if it meant we could go with the better technical solution, if I had the power to do that.
All that aside it's certainly not quite as clear cut as any language. Many languages have a mix of ascii and non-ascii characters and those fare better than they would with utf-16. And in the codepoints below 0x07ff (where utf-8 goes up to 3 bytes) there are character sets like greek, hebrew, arabic, and cyrillic. And as has been pointed out, ascii characters are often used in structural elements of documents and this factors into making it unlikely that the general case is as bad as the worst case (50% inflation) for most practical text. Particularly when you throw compression into the mix.
[edit] fixed some minor things.
Looking at the format definition[1] anything below U+0800 will fit in 2 bytes in UTF8. So the ethnocentrism starts at Samaritan[2]. The Japanese, Chinese and Indian scripts are probably the most important in that range.
The comment you're replying to says right there, "implementations are routinely broken because the corner cases are rarer" utf-16 misleads people into using fixed-width code where it won't work right.
>Applications that use Asian languages take a big hit from utf-8.
Not really. It's a moderate hit for pure text, but how often do you see bulk pure text, without any markup, and that can't be compressed?
This is exactly my point: lots of code also treats utf-8 as if it were ASCII.
> how often do you see bulk pure text, without any markup, and that can't be compressed?
That's a good argument. Thanks :)
Two things to this:
- In many cases this is an entirely non-destructive mistake to make. This is an advantage of utf-8, that intermediaries that only care about ascii characters (even in the presence of extended characters) can work with it.
- It will become readily apparent that you've got a problem on the first attempt to port your app to any language with a non-ascii script. With UTF-16 it only becomes apparent when you go beyond the BMP.
It's about time we actually start pressing for the idea that utf-16 was a terrible idea and that utf-8 should be the dominant wire format for unicode, with ucs4 if you really need to have a linear representation.
Utf-16 is confusing, complicated, and implementations are routinely broken because the corner cases are rarer. I really hope we're not still stuck with it in 50 years.