News for hackers. Stuff that matters.

A comment on UTF-8 good, UTF-16 bad. Parent comment

Re:UTF-8 good, UTF-16 bad by Locke1689 on Tuesday February 08, 2011 @01:28PM (#2194099)
No, that's the correct behavior. list only incidentally returns a single character in ASCII strings -- it's not required to. You shouldn't be using list on raw unicode strings.

  u'\U00010000'.encode('utf-8')
should produce the same result on every Python version.
[ Parent | on HN ]
Re:UTF-8 good, UTF-16 bad by ot on Wednesday February 09, 2011 @06:32AM (#2197116)
> You shouldn't be using list on raw unicode strings.

Why? I am using list only to show what are the values of s[0] and s[1].

What I am saying is that it returns the list of characters of the underlying representation, so a list of wide chars (possibly surrogate) if compiled with UTF16 or a list of 32bit characters if compiled with UTF16.

Are you suggesting that all the string processing (including iteration) should be done on a str encoded in UTF8 instead of using the native unicode type?

[ Parent | on HN ]