Re:Memory use in CPython and MicroPython
by patrec on Sunday July 02, 2017 @07:43PM
(#14684907)
What's your use case for iterating or indexing into code points?
[ Parent
| on HN ]
Re:Memory use in CPython and MicroPython
by ubernostrum on Monday July 03, 2017 @04:20AM
(#14686733)
My use case is that I don't want iterating/indexing to yield the underlying byte representation, since that may end up slicing up a codepoint large enough to require multiple bytes.
I want the fundamental constituent unit of my strings to be the thing that the string is conceptually made of. When I'm ready to turn it into a sequence of bytes, I'll do that myself.
But the text is not conceptually made of unicode code points. It's conceptually made of characters. Codepoints are basically ever only used for either
a) low-level mucking around like normalization -- a fringe use
b) a (in the general case) bad approximation to characters -- the normal use, which will produce bad results for various "atypical" cases
Now, personally, I'd often rather muck around with bytes because frequently they are a more useful abstraction than codepoints and lots of in practice important text operations can be performed quite fine (and efficiently) on them, even if the text is full of non-ascii characters.