News for hackers. Stuff that matters.
Re:Unicode In Python, Completely Demystified by ludwigvan on Saturday February 26, 2011 @10:39AM (#2265721)
The main trick is to remember is, to get utf-8, you have to encode, not decode. When I didn't know how unicode works that well, I mistakenly assumed unicode is coded, so I have to decode it. Internally, unicode is of course coded (that is represented by some coding), but you don't have to know it. As the slides say, just encode to utf8 when you print text out, or display it in a web browser etc; and you will be fine.

One other unicode library I can recommend is icu, Python bindings to IBM's ICU. It solved some Turkish specific problems I have. (Turkish alphabet has I,İ, ı, i which makes upper-lower case conversions tricky. For example, mayıs becomes MAYIS when capitalized while ENGLISH becomes englısh when lowercased.)

Read http://en.wikipedia.org/wiki/Turkish_dotted_and_dotless_I , http://www.joelonsoftware.com/articles/Unicode.html and http://www.codinghorror.com/blog/2008/03/whats-wrong-with-tu... for more info.

Another issue, if you are doing web dev is, how to represent non-ascii characters in urls. One choice is url encoding, the other is slugifying the url, that is, choosing an ascii equivalent for the non-ascii character. For example, Django has a slugify function that helps with this. It converts, for example, über to uber.

Re:Unicode In Python, Completely Demystified by zepolen on Saturday February 26, 2011 @11:59AM (#2265904)
>It converts, for example, über to uber.

What does it convert 'ぬびばざべ' into?

Re:Unicode In Python, Completely Demystified by admp on Saturday February 26, 2011 @01:21PM (#2266111)
ICU transliterator is fairly sophisticated (and configurable), actually. Take a look at the examples:

  キャンパス -> kyanpasu
  Αλφαβητικός Κατάλογος -> Alphabētikós Katálogos
  биологическом -> biologichyeskom
From: http://userguide.icu-project.org/transforms/general

Edit: reread, OP was talking about Django transliteration, which is much simpler.

Re:Unicode In Python, Completely Demystified by ludwigvan on Saturday February 26, 2011 @03:15PM (#2266405)
Yes, Django slugifier turns ぬびばざべ to an empty text. Maybe that's because my language is set to tr-tr, I am not sure. One possible downside with icu is that, afaik, you can't run it in Google App Engine.