News for hackers. Stuff that matters.

A comment on Brian's Rules for Writing Cross Platform 'C' Code (2008). Parent comment

Re:Brian's Rules for Writing Cross Platform 'C' Code (2008) by mtdewcmu on Saturday June 14, 2014 @11:53PM (#7894322)
That helps put some things in perspective. Unicode is not a well-understood topic among programmers, I've noticed (at least in the US), and for something so important it's surprisingly difficult to find information. The distinction between UCS-2 and UTF-16 must have made sense at one time, but it makes no sense anymore, and it's not necessarily easy to track down the explanation for why there are two 16-bit formats. Thanks.
[ Parent | on HN ]
Re:Brian's Rules for Writing Cross Platform 'C' Code (2008) by mikeash on Sunday June 15, 2014 @12:23AM (#7894364)
The whole thing is pretty much incomprehensible without understanding the history. It's a bit crazy how much historical baggage Unicode has managed to accumulate in just a couple of decades (and most of it from just the first few years).

It sounds like you have it figured out, but just in case: UCS-2 is the original pure 16-bit encoding from the days when nobody could conceive of needing more than 65536 characters for anything. UTF-16 is the hacked up "oops, it turns out there are a lot more Chinese characters than we realized" version that tries to maintain some compatibility with software built for UCS-2.

Given that there was only a five-year period between the first release of Unicode and the introduction of UTF-16, it's kind of amazing how pervasive the 16-bit encoding got. It's thoroughly baked into Java, where "char" was supposed to be a unicode character, but ended up being a unicode character or a piece of a UTF-16 surrogate pair. Windows APIs and Cocoa's NSString went through the same process. JavaScript too, and the spec still allows implementations to only support UCS-2.

[ Parent | on HN ]
Re:Brian's Rules for Writing Cross Platform 'C' Code (2008) by mtdewcmu on Sunday June 15, 2014 @12:54AM (#7894441)
The c and c++ apis seem to make things worse. The right solution for charset conversion seems to be iconv.
[ Parent | on HN ]