Re:Unicode In Python, Completely Demystified
by zepolen on Saturday February 26, 2011 @11:59AM
(#2265904)
>It converts, for example, über to uber.
What does it convert 'ぬびばざべ' into?
Re:Unicode In Python, Completely Demystified
by admp on Saturday February 26, 2011 @01:21PM
(#2266111)
ICU transliterator is fairly sophisticated (and configurable), actually. Take a look at the examples:
キャンパス -> kyanpasu
Αλφαβητικός Κατάλογος -> Alphabētikós Katálogos
биологическом -> biologichyeskom
From: http://userguide.icu-project.org/transforms/generalEdit: reread, OP was talking about Django transliteration, which is much simpler.
Re:Unicode In Python, Completely Demystified
by ludwigvan on Saturday February 26, 2011 @03:15PM
(#2266405)
Yes, Django slugifier turns ぬびばざべ to an empty text. Maybe that's because my language is set to tr-tr, I am not sure. One possible downside with icu is that, afaik, you can't run it in Google App Engine.
One other unicode library I can recommend is icu, Python bindings to IBM's ICU. It solved some Turkish specific problems I have. (Turkish alphabet has I,İ, ı, i which makes upper-lower case conversions tricky. For example, mayıs becomes MAYIS when capitalized while ENGLISH becomes englısh when lowercased.)
Read http://en.wikipedia.org/wiki/Turkish_dotted_and_dotless_I , http://www.joelonsoftware.com/articles/Unicode.html and http://www.codinghorror.com/blog/2008/03/whats-wrong-with-tu... for more info.
Another issue, if you are doing web dev is, how to represent non-ascii characters in urls. One choice is url encoding, the other is slugifying the url, that is, choosing an ascii equivalent for the non-ascii character. For example, Django has a slugify function that helps with this. It converts, for example, über to uber.