News for hackers. Stuff that matters.
Re:Python 3 makes it easier to develop high-quality software by 1st1 on Thursday November 19, 2015 @10:13AM (#10595173)
UTF-8 is just an encoding. Unicode is a more general term. Python 3 allows you to convert bytes to unicode from any encoding, including utf-8 (which is default). Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow.

For regexes, \w matches any unicode word character (unless ASCII mode is on), just use it on strings. If you use it on bytes, then yes, you can only match bytes.

Re:Python 3 makes it easier to develop high-quality software by Avernar on Thursday November 19, 2015 @01:35PM (#10596570)
> Using utf-8 as the internal unicode representation is flawed too - index and length operations are slow.

And this perfectly illustrates what I posted in another comment elsewhere. Thinking that doing operations on code points (indexing and length) is the proper thing to do. You need to use a unicode library to work on graphemes and take into account grapheme clusters.

> For regexes, \w matches any unicode word character

From what I've read, \w just matches code points. I did find that the https://pypi.python.org/pypi/regex module supports \X which matches graphemes as per the unicode spec http://www.unicode.org/reports/tr29/.