The last comment is not justified by the paper. The paper shows one case where the dumb thing finds tons of crashes, but several other cases in the same figure where it finds zero crashes. And the figure there is just one target program. More experiments on this issue might be interesting.

One other thing to point out: Finding crashes and finding bugs are not the same thing, as section 7 carefully argues. It could well be that many of those crashes are duplicates. The blog post I linked above also summarizes this point.

I didn't initally understand this from your summary, so trying to rephrase it a bit to how I understand it after glancing over the paper:

The paper's thesis is that most (?) currently claimed advances in fuzzing algorithms were not proven correctly enough (a.k.a. "in a statically rigorous/significant way") to actually be advances. Meaning, they may well have been just random fluctuations. Then, the paper proposes how to correctly compare fuzzers, and what the community needs to be able to do that (e.g. an agreed upon common body of codebases to be fuzzed, a set of known fuzzable bugs in those codebases, minimum fuzzing time, etc.)

> It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous.

Shall I tell you a not so well hidden secret? Almost nothing published in computer science, let alone in IT security, is statistically rigorous.

That's not true; plenty of papers in the systems field perform careful and rigorous experiments. This doesn't even include the theory side of stuff, where you only have proofs, and no statistics to get wrong.

Fuzzers don't have to be non-deterministic.