And even more so, if the current 20.1-bit format turns out to be insufficient, UTF-8 is happy to expand to at least 31 bits, where UTF-16 would be forced to undergo very awkward redesign, such as dedicating an entire plane to double-surrogates.

The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8. UTF-16 does not suffer from that particular problem.

Of course, there's a long ways to go from "a sequence of valid code units" to "a valid string," but it's still relevant.

>UTF-16 does not suffer from that particular problem

what about surrogate pairs? You can't have only one 16 bit word for a pair and have a valid UTF-16 sequence. This problem is real easy to do if you substring a UTF-16 sequence naively.

Right, there's still plenty of ways you can screw up a string. My point was merely that UTF-8 has one failure mode (invalid code units) that UTF-16 does not suffer from, and that has implications in encoding converters and APIs.

Invalid bytes is a feature. It makes it much easier to figure out the bytes you have are not actually UTF8.

UTF-8 is already much better than the legacy double-byte encodings (Big5, GB2312, Shift-JIS etc etc) because, at least you can tell from any byte whether it is supposed to be a part of a code sequence, without having to start all the way from the beginning of the string.

> The flip side of this is that there is such a thing as invalid UTF-8, i.e. a sequence of bytes that does not form a sequence of valid code units and so cannot be decoded as UTF-8.

In what sort of situation that is a problem?

It can allow for security exploits, if the different layers of your stack handle invalid code units differently.

There's a related issue with "non-shortest forms:" it's possible to encode the NUL character '\0' without actually using a null byte. This means that (for example) "round tripping" UTF-8 data can introduce a null byte, allowing for security exploits when mixed with C-style string processing.

This is untrue, see D79 and D89 of Unicode 6.2: an ill-formed UTF-16 string is one containing lone surrogates.

Yes, that's an ill-formed string. As I wrote, there's a long ways to go from a "sequence of valid code units" to "a valid string."

But what I wrote is nevertheless correct, and matters. See for example the special requirements on UTF-8 processing in D93:

    If the converter encounters an ill-formed UTF-8
    code unit sequence...it must not consume the
    successor bytes as part of the ill-formed
    subsequence whenever those successor bytes
    themselves constitute part of a well-formed UTF-8
    code unit subsequence....For a UTF-8 conversion
    process to consume valid successor bytes is not
    only non-conformant, but also leaves the converter
    open to security exploits."

"Ill-formed code unit sequence" is not possible with UTF-16. That's an advantage. Also note the last line: the existence of ill-formed code units has security implications that are not present with UTF-16.

To be fair, UTF-16 has its own unique issues too, like endianness.