News for hackers. Stuff that matters.
Re:Why Python 3 Exists by klodolph on Thursday December 17, 2015 @02:43PM (#10753920)
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32).

    >>> import sys
    >>> def charsize(x):
    ...     return sys.getsizeof(x + x) - sys.getsizeof(x)
    ...
    >>> charsize(b'A')
    1
    >>> charsize('A')
    1
    >>> charsize('\xff')
    1
    >>> charsize('\uffff')
    2
    >>> charsize('\U0010FFFF')
    4
Re:Why Python 3 Exists by the_mitsuhiko on Thursday December 17, 2015 @03:51PM (#10754442)
> But there are space optimizations on the Python string object which make it so it doesn't waste that much space

Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.

Re:Why Python 3 Exists by aidenn0 on Tuesday December 22, 2015 @07:03PM (#10781192)
Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.