News for hackers. Stuff that matters.
Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @01:57PM (#10753571)
Also, it's incompatible with UTF-8 strings stored in C, which means that when you cross the Python/C API boundary, you have to re-encode all strings. This is a large performance penalty right at the time when you can least afford performance penalties.

IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.

Re:Why Python 3 Exists by viraptor on Friday December 18, 2015 @12:31AM (#10756835)
> Also, it's incompatible with UTF-8 strings stored in C

That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/

So unless you go outside of UCS-1, you don't need to reencode anything.

Re:Why Python 3 Exists by shoyer on Thursday December 17, 2015 @02:29PM (#10753802)
This design doc from DyND (a possible NumPy alternative) has some useful references on this point: https://github.com/libdynd/libdynd/blob/master/docs/string-d...
Re:Why Python 3 Exists by rurban on Thursday December 17, 2015 @06:54PM (#10755622)
Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit. Very common on every better VM.

This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.

Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.

And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.

Re:Why Python 3 Exists by 0x0 on Thursday December 17, 2015 @08:34PM (#10756032)
"Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)
Re:Why Python 3 Exists by Matthias247 on Friday December 18, 2015 @03:53AM (#10757435)
However C also did not agree on UTF8. There are lots libraries which use UTF16 encoding, so you would need to convert for these if you stick to UTF8. And you maybe need some conversion anyway, because one system expects zero terminated strings while another expects explicit length fields. Unfortunately strings are really a mess.