News for hackers. Stuff that matters.
Re:Show HN: Faster UTF-8 validator by nitwit005 on Friday November 15, 2019 @11:12PM (#21550969)
There's not as big a niche as you might think. A lot of the software that needs valid UTF-8 don't validate upfront, but do it as part of a parsing process. An example would be something like an XML or JSON parser.

For more mundane uses like opening a big text file, you often want to tolerate invalid bytes and show a replacement character. You could use the technique, but you'd have to modify it to work for that usage.

Re:Show HN: Faster UTF-8 validator by zwegner on Saturday November 16, 2019 @05:10PM (#21555283)
In case anyone sees this: I initially agreed with you, thinking that pure UTF-8 validation is a bit more of a niche than might be expected. But if I'm not mistaken, I think at least for JSON validation, the UTF-8 validation can happen completely independently of the parsing. I don't think any control characters (braces, commas, quotes, escapes, etc) will change the validity of the UTF, or vice versa. So it might be beneficial (both for speed and security, as the sibling comment notes) to validate chunks of input as UTF-8, then pass them off to the parser, which now doesn't need to deal with validation.

Maybe this could be applied to XML, but that's quite a behemoth of a standard--I really wouldn't be surprised if there was a way to switch encodings mid-stream. I have no idea though...

Re:Show HN: Faster UTF-8 validator by nitwit005 on Tuesday November 19, 2019 @10:58PM (#21580973)
Sure, you can validate and then parse the JSON, but parsing it is also effectively validating it as UTF-8, except for the string values in the JSON, so it's heavily going to be duplicated effort. And unfortunately, the strings can contain things like invalid surrogate pairs by using unicode escapes, so you may want to validate after escaping.

You can't normally blindly parse XML as UTF-8. The encoding has rules for detecting the character set.

Re:Show HN: Faster UTF-8 validator by pdkl95 on Saturday November 16, 2019 @12:58AM (#21551267)
[re: upfront parsing - a PSA about validating input]

> A lot of the software ... don't validate upfront

They almost always should be validating all input up front. Deferring validation tends to become many different pieces of code all informally parsing fragments of the input that are needed locally. Without upfront validation of the complete unit of input, the resulting fragmented parsers are just a weird machine waiting to be programmed by a malicious attacker.

For a much better explanation, I strongly recommend Meredith and Sergey's 28c3 talk[1] about The Science of Insecurity.

> you often want to tolerate invalid bytes and show a replacement character

While this validator wouldn't be useful in that situation, it is still important to validate the input upfront. When showing replacement characters, the "invalid bytes" that will be supported with a replacement character should be formally defined and added to the validation grammar, because they are no longer "invalid", but instead will be handled as a special case.

Postel's robustness principle shouldn't be used as an excuse to skip validation. Being "liberal in what you accept" should still be well-defined and validated.

[1] https://media.ccc.de/v/28c3-4763-en-the_science_of_insecurit...

Re:Show HN: Faster UTF-8 validator by nitwit005 on Tuesday November 19, 2019 @11:03PM (#21580990)
You trimmed my quote in a way that completly changes the meaning, so I'm not even going to bother responding to that.

How unicode replacement characters should work is already well defined. Just look it up.