What's your use case for iterating or indexing into code points?
Re:Memory use in CPython and MicroPython
by ubernostrum on Monday July 03, 2017 @04:20AM
(#14686733)
My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes.
I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.
But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either
a) low-level mucking around like normalization -- a fringe use
b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases
Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.
I do, and I do believe Python 3's approach is superior for strings.
An array of 8-bit integers is not a string. It's an array of 8-bit integers. I want my string type to be a sequence of Unicode codepoints, and I want iterating it to yield codepoints, I want indexing into it to yield the codepoint at the index, and I want the length of it to be the number of codepoints it contains. When I'm ready to send it over a wire or flush it to disk, then I'll think about the best way to represent it as an array of 8-bit integers, but bringing the baggage of encodings (especially the baggage of UTF-8) into any part of my code except the boundary layer is asking for trouble.