Re:Why Python 3 Exists
by klodolph on Thursday December 17, 2015 @02:43PM
(#10753920)
You're right that O(1) access is pointless, and UTF-8 is fine. But there are space optimizations on the Python string object which make it so it doesn't waste that much space. Basically, it dynamically selects LATIN-1, UCS-2, or UTF-32 (or you could call them UCS-1, UCS-2, UCS-4, but there isn't really any such thing as UCS-1, and UCS-4 is an obsolete name for UTF-32).
[ Parent
| on HN ]
Re:Why Python 3 Exists
by the_mitsuhiko on Thursday December 17, 2015 @03:51PM
(#10754442)
> But there are space optimizations on the Python string object which make it so it doesn't waste that much space
Except it does not help. When you render out a template it starts out as latin1, then the first time it hits a unicode char the whole thing re-encodes into ucs2 and then when it sees an emoji it re-encodes to ucs4. In practical terms you will see UCS2 and UCS4 all the time.
Except UTF-8 is never less compact than UTF-32 (it is one to four octets, and UTF-32 is always four octets), and is often more compact than UTF-16, with the sole exception being BMP CJK text that is not embedded in an ascii-based markup language.