This comes up because many implementations of "UTF-16" are in fact naive, and just operate within the mentality of UCS-2, where everything is just a 16-bit number.

This is relevant within this discussion because we were talking about filesystems, and Windows stores file names as an arbitrary array of 16-bit numbers, but UTF-16 has additional requirements that Windows file names do not satisfy.

When I say UCS-2, I just mean pretending that UTF-16 is short for `[]uint16`, and not worrying about surrogate pairs.

I think we can all agree that you should never be applying any normalization to bytes representing a file system path, unless you are doing it according to specifications for that particular file system+OS combo.

file system paths are tricky with non-ascii, that is for sure! Especially if you are trying to be agnostic to OS/filesystem. They are an edge case.

UCS-2 refers to the UTF-16 encoding of the first plane, defined when there was only one plane's worth of assignable space. UTF-16 extends UCS-2, and defines the reserved surrogates, thereby introducing the possibility of invalid surrogate usage. The parent post is referring to that (so it would be more accurate for the parent post to have said "UTF-16").

OK, thanks.

You can certainly have invalid bytes for UTF-8 too. I think in any unicode encoding, not every possible byte sequence is a valid representation of unicode codepoints. It doesn't require UCS-2/UTF-16.

Here's an invalid byte sequence in UTF-8: "\xc3\x28"

True, though the GP's example was somewhat unique in that UTF-16 created the possibility of invalid combinations at the code point level, rather than only the byte level.

> I'm not certain, I'm not familiar with that situation.

See RFC 6531.