News for hackers. Stuff that matters.
Re:Why Python 3 Exists by twoodfin on Thursday December 17, 2015 @01:22PM (#10753240)
Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading:

https://github.com/python/cpython/blob/master/Objects/unicod...

Re:Why Python 3 Exists by nostrademons on Thursday December 17, 2015 @01:57PM (#10753571)
Also, it's incompatible with UTF-8 strings stored in C, which means that when you cross the Python/C API boundary, you have to re-encode all strings. This is a large performance penalty right at the time when you can least afford performance penalties.

IMNSHO, most modern languages should be storing strings as UTF-8 and give up on random access by characters. You almost never need it; in the most frequent case where you do (using indexOf or equivalent to search for a substring, and then breaking on it), you can solve the problem by returning a type-safe iterator or index object that contains a byte offset under the hood, and then slicing on that. Go, Rust, and Swift have all gone this route.

Re:Why Python 3 Exists by viraptor on Friday December 18, 2015 @12:31AM (#10756835)
> Also, it's incompatible with UTF-8 strings stored in C

That's not true as of 3.3. It will store the compact representation, which in most cases is ASCII, which is UTF-8 compatible. https://www.python.org/dev/peps/pep-0393/

So unless you go outside of UCS-1, you don't need to reencode anything.

Re:Why Python 3 Exists by shoyer on Thursday December 17, 2015 @02:29PM (#10753802)
This design doc from DyND (a possible NumPy alternative) has some useful references on this point: https://github.com/libdynd/libdynd/blob/master/docs/string-d...
Re:Why Python 3 Exists by rurban on Thursday December 17, 2015 @06:54PM (#10755622)
Nope. This is just a normal short-string implementation, inlined with 64bit systems, with a tag bit. Very common on every better VM.

This has nothing to do with (inefficient) encodings, and their bad historic namings, probably derived from perl.

Of course any name like byte2utf8 or just to_byte or b2u8 would have been better than encode/decode.

And of course cache size matters nowadays much more than immediate substring access for utf8, so nobody should use ucs-2 or even ucs-4 internally. This is easily benchmarkable.

Re:Why Python 3 Exists by 0x0 on Thursday December 17, 2015 @08:34PM (#10756032)
"Byte2utf8" is a pretty confusing name for a method, considering utf8 is a byte encoding of unicode... :)
Re:Why Python 3 Exists by Matthias247 on Friday December 18, 2015 @03:53AM (#10757435)
However C also did not agree on UTF8. There are lots libraries which use UTF16 encoding, so you would need to convert for these if you stick to UTF8. And you maybe need some conversion anyway, because one system expects zero terminated strings while another expects explicit length fields. Unfortunately strings are really a mess.
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @01:48PM (#10753472)
> Indeed, the Python 3 Unicode string object is fascinatingly clever. Code worth reading:

Clever is relative. The Python 3 unicode string objects makes very little sense given real world situations. It wastes enormous amount of memory in many common scenarios for no reason at all other than to facilitate O(1) access to charpoints which is completely inappropriate for text processing anyways.

Worse than that: it can blow up by another 50% the moment someone encodings it to utf-8 as the encoded version is cached around.

The object is too clever for its own good.

Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @02:43PM (#10753920)
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32).

    >>> import sys
    >>> def charsize(x):
    ...     return sys.getsizeof(x + x) - sys.getsizeof(x)
    ...
    >>> charsize(b'A')
    1
    >>> charsize('A')
    1
    >>> charsize('\xff')
    1
    >>> charsize('\uffff')
    2
    >>> charsize('\U0010FFFF')
    4
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @03:51PM (#10754442)
> But there are space optimizations on the Python string object which make it so it doesn't waste that much space

Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.

Re:Why Python 3 Exists by aidenn0 on Tuesday December 22, 2015 @07:03PM (#10781192)
Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.
Re:Why Python 3 Exists by Animats on Friday December 18, 2015 @12:34AM (#10756843)
Another alternative would have been to store strings in UTF-8 and generate an array of indices to the starting point of each entity only if needed for random access. Subscripting a string would use the array of indices.

Only a few standard member functions of string (find, rfind, index, rindex) return integer subscripts into a string. Those could return an opaque index object which has an index and a link to the string. Such objects could be used for string operations which need a string index. If an index object is forcibly converted to an integer, the array of indices has to be generated, but otherwise, it is unnecessary. You could even allow adding or subtracting integers from an index object, which would walk the UTF-8 string forwards or backwards as indicated.

This would make UTF-8 strings usable with subscript and slice notation without exploding them to wide characters.

Re:Why Python 3 Exists by eugenekolo2 on Thursday December 17, 2015 @11:40PM (#10756682)
Yeah, I'm not reading 15K LOC... What makes it clever? What is even the point of linking a 15K file?