News for hackers. Stuff that matters.
Re:Python 3 makes it easier to develop high-quality software by Avernar on Wednesday November 18, 2015 @04:32PM (#10590903)
Yes it is. Every conversion to and from the unicode type and every expansion of the unicode type from 1 to 2 to 4 bytes eats up processor time, cache/memory bandwidth and puts more pressure on the garbage collector.

And I don't want to be forced to distinguish between strings and bytes. To me they are one and the same as I use UTF-8 for everything. Not once in my code do I need to encode/decode anything.

Re:Python 3 makes it easier to develop high-quality software by Avernar on Thursday November 19, 2015 @01:26AM (#10593089)
Nice, down voted for someone not agreeing with me. I'd rather know what you though was wrong with what I wrote.

I've always wanted to know how things are done internally. Makes me a better programmer. What I said about all this extra conversion does to performance is correct, especially when scaling up.

And my choice to use UTF-8 internally is a valid way to do things. I make sure that what comes in from the outside world is UTF-8 (either by rejecting or converting) and then don't have to worry about it from that point on. The bulk of my code deals with outbound data so no need for me to do any encoding as it's already UTF-8.

Re:Python 3 makes it easier to develop high-quality software by mark-r on Wednesday November 18, 2015 @06:17PM (#10591459)
Most of us aren't so lucky. I really do wish we lived in a 100% UTF-8 world. But even if we did, I don't see how e.g. regular expressions would work.
Re:Python 3 makes it easier to develop high-quality software by Avernar on Thursday November 19, 2015 @01:13AM (#10593044)
The world seems to be headed to UTF-8 for everything. That's why having Python3 force us away from it doesn't make sense to me.

As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.

Re:Python 3 makes it easier to develop high-quality software by 1st1 on Thursday November 19, 2015 @10:13AM (#10595173)
UTF-8 is just an encoding. Unicode is a more general term. Python 3 allows you to convert bytes to unicode from any encoding, including utf-8 (which is default). Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow.

For regexes, \w matches any unicode word character (unless ASCII mode is on), just use it on strings. If you use it on bytes, then yes, you can only match bytes.

Re:Python 3 makes it easier to develop high-quality software by Avernar on Thursday November 19, 2015 @01:35PM (#10596570)
> Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow.

And this perfectly illustrates what I posted in another comment elsewhere. Thinking that doing operations on code points (indexing and length) is the proper thing to do. You need to use a unicode library to work on graphemes and take into account grapheme clusters.

> For regexes, \w matches any unicode word character

From what I've read, \w just matches code points. I did find that the https://pypi.python.org/pypi/regex module supports \X which matches graphemes as per the unicode spec http://www.unicode.org/reports/tr29/.