Re:How fast can you validate UTF-8 strings in JavaScript?
by chubot on Saturday January 20, 2024 @04:28PM
(#39072546)
I would like to see some real examples too, but I think a pretty common thing is bypassing URL deny-lists with different representations of the same string, or any kind of ACL
e.g. the developer might either deny or allow /admin/, or /admin/.*
But if a given Unicode string [1] has two representations, then that check may not work as intended for one of them.
---
UTF-8 represents code points with 1-4 bytes, which are roughly similar to base 64 digits (because a continuation byte is 10xx_xxxx, it has 6 bits of freedom)
This opens up the possibility of "overlong encodings", which a UTF-8 validator must reject.
That is, you can do the equivalent of representing a number as "09" instead of "9" -- that's an overlong encoding. A decoder may understand "09" and "9" as both being the digit 9, but it's a different sequence of bytes.
So a naive decoding algorithm can accept a spelling of "admin" as 6, 7, 8, ... 20 bytes, not 5. There are many overlong encodings!
The only valid UTF-8 spelling has 5 bytes, but decoders that don't validate will "naturally" accept more (in fact I think I even wrote one of these :-/ )
In summary, if you don't do UTF-8 validation, then one layer doesn't reject the invalid representation, and another layer may decode it into a valid Unicode string.
Also, some regex engines work on UTF-8 encoded bytes, and some work on arrays of code points.
---
Again I'd be interested in more real examples.
[1] A sequence of "Unicode scalars" -- code points not in the surrogate range
e.g. the developer might either deny or allow /admin/, or /admin/.*
But if a given Unicode string [1] has two representations, then that check may not work as intended for one of them.
---
UTF-8 represents code points with 1-4 bytes, which are roughly similar to base 64 digits (because a continuation byte is 10xx_xxxx, it has 6 bits of freedom)
This opens up the possibility of "overlong encodings", which a UTF-8 validator must reject.
That is, you can do the equivalent of representing a number as "09" instead of "9" -- that's an overlong encoding. A decoder may understand "09" and "9" as both being the digit 9, but it's a different sequence of bytes.
So a naive decoding algorithm can accept a spelling of "admin" as 6, 7, 8, ... 20 bytes, not 5. There are many overlong encodings!
The only valid UTF-8 spelling has 5 bytes, but decoders that don't validate will "naturally" accept more (in fact I think I even wrote one of these :-/ )
In summary, if you don't do UTF-8 validation, then one layer doesn't reject the invalid representation, and another layer may decode it into a valid Unicode string.
Also, some regex engines work on UTF-8 encoded bytes, and some work on arrays of code points.
---
Again I'd be interested in more real examples.
[1] A sequence of "Unicode scalars" -- code points not in the surrogate range