News for hackers. Stuff that matters.
Re:How to break everything by fuzz testing by pfdietz on Monday April 27, 2020 @04:22AM (#22993964)
A very funny thing about fuzzing is how random input testing used to be so looked down upon by the software testing community. Read old (1970s) testing books and you'll see comments like "random testing is the worst kind of testing". I still saw this even as recently as a decade ago.
Re:How to break everything by fuzz testing by praptak on Monday April 27, 2020 @05:02AM (#22994165)
Fuzzing is not random testing though. It's directed random testing.
Re:How to break everything by fuzz testing by pfdietz on Monday April 27, 2020 @05:39AM (#22994333)
The original fuzzing was as random + black box as it gets.
Re:How to break everything by fuzz testing by UncleMeat on Monday April 27, 2020 @11:12AM (#22996591)
Yes, and things like coverage guided fuzzing have completely revolutionized things. Prior to directed fuzzing, it was okay but largely unimpressive. Now it blazes through code structures that were previously used as motivating examples for symbolic execution. It is a meaningfully different technique today.
Re:How to break everything by fuzz testing by pfdietz on Monday April 27, 2020 @12:08PM (#22997144)
Well, that depends on what you mean by "impressive", and in what domain. Black box compiler fuzzing has been very effective.
Re:How to break everything by fuzz testing by UncleMeat on Monday April 27, 2020 @01:25PM (#22997875)
That's actually one of the few fields where I feel like fuzzing has underperformed. There was an interesting paper at OOPSLA this year that found that while the fuzzing community has indeed found a lot of bugs, that these bugs actually are triggered by real code approximately-never. It was a really interesting result coming from within a community that ordinarily biases towards overinflating the value of PL techniques.
Re:How to break everything by fuzz testing by pfdietz on Monday April 27, 2020 @01:39PM (#22998016)
That paper, if it's the one I'm thinking of, found that a compiler bug found by fuzzing was more likely to be found by a user, than a user found bug was likely to be found by another user. So if fuzzing-found bug reports are bad, user-found bug reports are even less useful.

Another thing to remember is that as blackbox fuzzing became state of the practice, its benefit declined, as the bugs it would find would be found early, by the developers themselves. All testing techniques are self-limiting this way.

I want you to look at the results of jsfunfuzz and tell me the impact of that wasn't profound.

Re:How to break everything by fuzz testing by m000 on Monday April 27, 2020 @08:10AM (#22995040)
That piece of advice was probably valid in the 1970s: The computers were far too slow and far too expensive for any kind of random testing to make sense. Fuzzing became popular when multi-core CPUs became commonplace and RAM more affordable.
Re:How to break everything by fuzz testing by dnautics on Monday April 27, 2020 @06:17AM (#22994513)
You really shouldn't do random testing. Fuzzing is better, but property testing (fuzzing with shrinkage) is even better.
Re:How to break everything by fuzz testing by pfdietz on Monday April 27, 2020 @09:44AM (#22995724)
The original prejudice was against any sort of randomness in testing. Manually constructed tests were seen as superior. That may have been true when computer time was dear, but the bias persisted into the latest edition of a well known book on software testing, published after (for example) Csmith had been released.