> Also, it's incompatible with UTF-8 strings stored in C
That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/
So unless you go outside of UCS-1, you don't need to reencode anything.
This design doc from DyND (a possible NumPy alternative) has some useful references on this point:
https://github.com/libdynd/libdynd/blob/master/docs/string-d...
Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit.
Very common on every better VM.
This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.
Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.
And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.
"Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)
Re:Why Python 3 Exists
by Matthias247 on Friday December 18, 2015 @03:53AM
(#10757435)
However C also did not agree on UTF8. There are lots libraries which use UTF16 encoding, so you would need to convert for these if you stick to UTF8. And you maybe need some conversion anyway, because one system expects zero terminated strings while another expects explicit length fields. Unfortunately strings are really a mess.
IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.