Re:OMG Ponies (Aka Humanity: Epic Fail)
by blasdel on Monday November 02, 2009 @10:39PM
(#918526)
Nope, Python fucks this up just the same, even in Python 3:
Almost no implementation will fuck up LATIN SMALL LETTER E WITH ACUTE U+00E9, but nearly all programming languages will royally fuck up a COMBINING ACUTE ACCENT U+0301 even in much easier cases like string length. Almost all implementations that claim to be UTF-16 are actually UCS-2, and can't handle surrogates in the slightest.
[ Parent
| on HN ]
You can get the correct result by normalizing the string first:
>>> import unicodedata
>>> print unicodedata.normalize('NFC', u'Les Mise\u0301rables')[::-1]
selbarésiM seLThat's in some sense even more broken: Not all combined characters have a normalized form, so the result is even less predictable. (I don't have an example ready.)
Here’s a good series of articles on how it works in Ruby: http://blog.grayproductions.net/categories/character_encodin...
Apparently Ruby 1.9 handles it well.
(I wouldn’t be surprised if this had previously been linked on HN.)