I can see how that could be a useful optimization for nodes already in UTF-8, but in my scheme, the rope would have to accommodate fixed-width codecs such as "raw" and "latin-1", too.
For instance, if you read bytes from a file with no encoding specified, then the resulting rope would be comprised of only "raw" nodes.
If it's PNG image data, then it wouldn't make sense to decode them as UTF-8 :-)
Re:Python's missing string type
by TheLoneWolfling on Friday April 18, 2014 @12:52PM
(#7609946)
Well, yeah. The above approach just means that you can (almost) always index in O(log(n)) time or better.
If you want to handle multiple encodings, you drop the length field entirely: you have an encoding field, with the different lengths of variable-length charsets internally being expanded out to different encodings.
(So, instead of having UTF-8, you have UTF-8-1-byte, UTF-8-2-byte, UTF-8-3-byte, and UTF-8-4-byte (as well as plain UTF-8). Probably internally mapped to a 16-bit enum or something. Again, this is all internal: to write out to a file or something you map it back to the underlying encoding.)
Use a rope type that's entirely (modified) UTF-8 throughout, but with the wrinkle that every character in a node must take the same number of bytes per character. (Use overlong encodings internally where appropriate, namely where you have (for example) 1 one-byte character in the middle of a long run of two-byte characters.)
You can even combine that approach with this.