First of all, if you don't know the encoding, then you don't know the encoding, and you will need to figure out if it's UTF-8, UTF-16, ISO-8859-1, etc. If you happen to know that it's UTF-16, you still need to figure out if it's UTF-16BE or LE.
> "It's not fixed width" But for most text, it is.
This is a dangerous way of thinking. One of the big problems with UTF-16 is that for most text, it is fixed width; so many people make that assumption, and you never notice the problem until someone tries to use an obscure script or an emoji character. This means that bugs can easily be lurking under the surface; while with UTF-8, anything besides straight ASCII will break if you assume fixed width, making it much more obvious.
> Sure, you can do UTF-32 and it may not be a bad idea (today)
UTF-32 isn't really meaningfully fixed width either. Sure, each code point is represented in a fixed number of bytes, but code points are not necessarily the interesting unit you want to index by. A glyph could be composed of several code points. Most of the time, you actually want to deal with text in longer units such as words or tokens, which are going to be variable width anyhow. The actual width of individual code points is only really of interest to low-level text processing libraries, not most applications.
True. But as you said, you have to know if it's BE or LE on UTF16. And there are ways to determine that automatically. (or it's on the same platform so it doesn't matter). With "ASCII compatible" codes, you can't.
I guess the main issue to me is that UTF-16 is not "ASCII compatible" so you know it's a different beast altogether.
And don't worry, I'm not assuming UTF-16 is fixed width. One should use the libraries and not try to solve this 'manually'.
About UTF-32 think: CPU registers and operations. Working with bytes is inefficient (even with the benefit of smaller size).
Yes, with UTF-16 you need to know not just the encoding, but also the endianness. That makes UTF-16 worse, not better.
UTF-16 is really the worst of all possible worlds: tons of wasted space with all the complexities of variable encoding without fixed endianness.
What does this mean/prove? I certainly hope you don't intend this to mean "so just pretend in your code that it will always be fixed width". And if that's not what you mean, then I don't know what you gain from that statement.
It is actually not that simple. By using UTF-16 you already have at least two problems:
1. You should know if byte order is big endian or little endian.
2. You should know if your API supports whole unicode set or only 65536 symbols. E.g. Windows API. Do you know answer? What will happen if your user wants to abuse your system by using symbols outside those 65536.
I did my first programming assignment on punched cards, so I probably have permanent ASCII/EBCDIC brain damage. However, this article decisively convinces even me that utf-8 wins and other encodings represent fail.
That obviously depends entirely on what kind of application we're talking about. Keeping large amounts of text data in memory as efficiently as possible is one of my greatest concerns. Many people are processing lots of text nowadays, more than ever before.
"It's not fixed width" But for most text, it is.
True, so ignoring it means that your code will be correct ... most of the time.
Depends on what you consider efficiency. If it's size, sure, store it using UTF-8. But if you're worried about speed, then UTF-16 or 32 may be the way to go, since you're dealing with data that fits a CPU register. For example, on ARM comparing one byte is much more work than comparing one 32-bit value.
"True, so ignoring it means that your code will be correct ... most of the time."
No, not going to ignore it! But on UTF-16 more code points match the UTF-16 encoding of it (easier for debugging)
"UTF-16 is the worst of both worlds—variable length and too wide"
Really, the author tries to convince the reader, but it's not that clean cut.
One of the advantages of UTF-16 is knowing right away it's UTF-16 as opposed to deciding if it's UTF-8/ASCII/other encoding. Sure, for transmission it's a waste of space (still, text for today's computer capabilities is a non issue even if using UTF-32)
"It's not fixed width" But for most text, it is. Sure, you can do UTF-32 and it may not be a bad idea (today)
Yes, Windows has to deal with several complications and with backwards compatibility, so it's a bag of hurt. Still, they went the right way (internally, it's unicode, period.)
"in plain Windows edit control (until Vista), it takes two backspaces to delete a character which takes 4 bytes in UTF-16"
If I'm not mistaken this is by design. The 4 byte characters is usually typed as a combination of characters, so if you want to change the last part of the combination you jut type one backspace.