Re:Python's missing string type
by TheLoneWolfling on Friday April 18, 2014 @12:52PM
(#7609946)
Well, yeah. The above approach just means that you can (almost) always index in O(log(n)) time or better.
If you want to handle multiple encodings, you drop the length field entirely: you have an encoding field, with the different lengths of variable-length charsets internally being expanded out to different encodings.
(So, instead of having UTF-8, you have UTF-8-1-byte, UTF-8-2-byte, UTF-8-3-byte, and UTF-8-4-byte (as well as plain UTF-8). Probably internally mapped to a 16-bit enum or something. Again, this is all internal: to write out to a file or something you map it back to the underlying encoding.)
If you want to handle multiple encodings, you drop the length field entirely: you have an encoding field, with the different lengths of variable-length charsets internally being expanded out to different encodings.
(So, instead of having UTF-8, you have UTF-8-1-byte, UTF-8-2-byte, UTF-8-3-byte, and UTF-8-4-byte (as well as plain UTF-8). Probably internally mapped to a 16-bit enum or something. Again, this is all internal: to write out to a file or something you map it back to the underlying encoding.)