News for hackers. Stuff that matters.
Re:Python 3 makes it easier to develop high-quality software by Avernar on Thursday November 19, 2015 @01:13AM (#10593044)
The world seems to be headed to UTF-8 for everything. That's why having Python3 force us away from it doesn't make sense to me.

As for regular expressions, they work perfectly if your delimiters are in the ASCII range. If you're trying to match a single character then they don't work for Unicode even when using Python3's unicode type. The reason is that code points are not characters.

Re:Python 3 makes it easier to develop high-quality software by 1st1 on Thursday November 19, 2015 @10:13AM (#10595173)
UTF-8 is just an encoding. Unicode is a more general term. Python 3 allows you to convert bytes to unicode from any encoding, including utf-8 (which is default). Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow.

For regexes, \w matches any unicode word character (unless ASCII mode is on), just use it on strings. If you use it on bytes, then yes, you can only match bytes.

Re:Python 3 makes it easier to develop high-quality software by Avernar on Thursday November 19, 2015 @01:35PM (#10596570)
> Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow.

And this perfectly illustrates what I posted in another comment elsewhere. Thinking that doing operations on code points (indexing and length) is the proper thing to do. You need to use a unicode library to work on graphemes and take into account grapheme clusters.

> For regexes, \w matches any unicode word character

From what I've read, \w just matches code points. I did find that the https://pypi.python.org/pypi/regex module supports \X which matches graphemes as per the unicode spec http://www.unicode.org/reports/tr29/.