Re:Fast UTF-8 validation
by josephg on Tuesday October 20, 2020 @02:58PM
(#24841107)
I think you’re confusing two concepts. There’s two meanings of “invalid” here - UTF-8 strings which contain invalid / unknown sequences, and strings which are packed in an invalid format. (Eg where the last byte is missing from a multi-byte code point).
I agree that usually you should often allow and preserve unknown code point sequences, because rendering degrades gracefully and Unicode keeps changing. But the algorithm this article is talking about checks that a UTF8 string is ‘valid’ in the context of the raw byte encoding format. These checks are much more simple, and they’re needed in basically any software that interacts with strings.
I agree that usually you should often allow and preserve unknown code point sequences, because rendering degrades gracefully and Unicode keeps changing. But the algorithm this article is talking about checks that a UTF8 string is ‘valid’ in the context of the raw byte encoding format. These checks are much more simple, and they’re needed in basically any software that interacts with strings.