Re:Unicode is, in 2012, still “cutting edge.”
by lambda on Wednesday July 11, 2012 @09:48AM
(#4229174)
Yeah, looks like Textmate is simply running the UTF-8 algorithm over UTF-16 code units, so each surrogate is being turned into a single UTF-8 code unit (which decodes to an invalid character).
It turns out that this is such a common mistake that there's even a name for this encoding, CESU-8: http://en.wikipedia.org/wiki/CESU-8
The same experiment with cat in place of textmate works fine, so it's textmate that is buggy.
According to "od -x1", textmate is writing:
So textedit is right to complain.