Re:Python 3 makes it easier to develop high-quality software
by Avernar on Wednesday November 18, 2015 @03:25PM
(#10590483)
> Recent versions only convert to UTF-32 when a wide character is present
It's non-BMP characters that trigger the conversion to 4 bytes per codepoint. Unfortunately Emoji are non-BMP and are super common.
Re:Python 3 makes it easier to develop high-quality software
by rspeer on Wednesday November 18, 2015 @04:35PM
(#10590916)
If you're dealing with emoji -- and yes, they are incredibly common -- you'd better not use any Python before 3.3.
Non-BMP characters in 2.x-3.2 give you implementation-specific behavior. It can vary from one system to the next. Some string methods fail on "narrow builds" of Python (the ones that store their characters in UCS-2), while others return different results than they would on a "wide build".
Re:Python 3 makes it easier to develop high-quality software
by Avernar on Wednesday November 18, 2015 @04:48PM
(#10591008)
Not an issue because I don't use the Unicode type. All my data comes in as UTF-8 on which I do a quick validation check. Then it's just handled as bytes all the way to and from my database.
Re:Python 3 makes it easier to develop high-quality software
by rspeer on Thursday November 19, 2015 @03:04AM
(#10593397)
Sure, that is an option that's always available. You don't have to worry if Python's string methods are wrong if you're opting not to use them.
Re:Python 3 makes it easier to develop high-quality software
by upofadown on Wednesday November 18, 2015 @11:55AM
(#10588919)
>Recent versions only convert to UTF-32 when a wide character is present (they also use 2 byte characters when that is sufficient) ...
I happen to know this fact. Are you really asking that everyone that comments on a language acknowledge all the possible implementation details? I at no point complained about the memory required to hold strings in CPython so I can't really understand why you would bring this up.
Re:Python 3 makes it easier to develop high-quality software
by maxerickson on Wednesday November 18, 2015 @12:00PM
(#10588963)
Because someone else reading "converting everything to UTF-32" might not be aware of the nuance. That's all.
Re:Python 3 makes it easier to develop high-quality software
by teach on Wednesday November 18, 2015 @12:24PM
(#10589174)
I'm a casual user of Python 2, and I certainly didn't know this, so I appreciated the clarification.
Re:Python 3 makes it easier to develop high-quality software
by upofadown on Wednesday November 18, 2015 @05:56PM
(#10591363)
Sorry for the snark but this has become a trigger for me. Every time anyone disagrees with the way Py3 does Unicode someone will immediately assume that that person is just confused about the implementation.
https://docs.python.org/3.3/whatsnew/3.3.html#pep-393-flexib...
Glancing at Julia, I'm not sure "invalid character index" is really a great thing to have to handle when dealing with text. It does seem that systems for dealing with text will eventually offer various apis beyond just a somewhat disfunctional character api (which I think is a fair description of str in Python 3).