You can do the same thing using VLQ, with less wastage. Just scan for the next cleared high bit.
Re:UTF-8 history (2003)
by mirekrusin on Thursday October 10, 2019 @06:23AM
(#21212772)
You can also construct malicious payloads which will DoS many implementations with architecture like that.
You can also trivially defend against that by having a max travel distance of, say, 4.
Re:UTF-8 history (2003)
by mirekrusin on Saturday October 12, 2019 @09:41PM
(#21237700)
Exactly, so you arrived at the same conclusion - fixed distance so you can safely pick up stream at arbitrary position.
The difference is that the distance limits would be inherent to the codec itself, rather than taking up extra space in the data stream.
Imagine a text editor displaying a large CSV file. Wouldn't you mind having to read everything betseen two locations if you jump forward from the one to the other? Or read everything from the start if you jumping backwards? Even if the text editor stores its own synchronization points, it has to read the file completely at least once, which can be annyoing for very large files.
Also, many of the simpler text tools that only look for ASCII bytes wouldn't work. For example, printing the last lines.
Also, I don't know that other sytem, but what about robustness if there's one bad byte somewhere in the file?