That helps put some things in perspective. Unicode is not a well-understood topic among programmers, I've noticed (at least in the US), and for something so important it's surprisingly difficult to find information. The distinction between UCS-2 and UTF-16 must have made sense at one time, but it makes no sense anymore, and it's not necessarily easy to track down the explanation for why there are two 16-bit formats. Thanks.

The whole thing is pretty much incomprehensible without understanding the history. It's a bit crazy how much historical baggage Unicode has managed to accumulate in just a couple of decades (and most of it from just the first few years).

It sounds like you have it figured out, but just in case: UCS-2 is the original pure 16-bit encoding from the days when nobody could conceive of needing more than 65536 characters for anything. UTF-16 is the hacked up "oops, it turns out there are a lot more Chinese characters than we realized" version that tries to maintain some compatibility with software built for UCS-2.

Given that there was only a five-year period between the first release of Unicode and the introduction of UTF-16, it's kind of amazing how pervasive the 16-bit encoding got. It's thoroughly baked into Java, where "char" was supposed to be a unicode character, but ended up being a unicode character or a piece of a UTF-16 surrogate pair. Windows APIs and Cocoa's NSString went through the same process. JavaScript too, and the spec still allows implementations to only support UCS-2.

The c and c++ apis seem to make things worse. The right solution for charset conversion seems to be iconv.

btw, the wchar_t typedef is 16 bits for MSVC, but 32 bits for gcc (even for MinG on Windows!). UTF-16 is the worst of all worlds.

Original author here....

This is why we do not use wchar_t as a cross platform type. We like "char *" with NULL terminated Utf8 strings, which works really well on all platforms. Unfortunately on Windows, some of the core API still take Utf16 (2 bytes per char). Since it is a Windows call it must be in an #ifdef anyway, and so IMMEDIATELY BEFORE calling into the Windows call at the last possible moment we convert the string from Utf8 to Utf16. It sounds like a pain, but after you have wrapped all your OS specific calls (like File I/O) you're calling the higher level anyway, and it was completely worth it.