I'm one of the authors of the paper. Two quick things I want to say:

I've written a blog post that is a short but rigorous description of the paper's results: http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical...

The paper does not make strong statements of the form, "fuzzers are/do X" or "all prior papers' claims are bogus." Rather, we say that the standard of evidence should be higher, and we demonstrate why failing to reach that standard could result in bogus claims. It's quite possible that a paper's idea really is an improvement. But it's also possible that additional evidence would cast doubt on that, or nuance it. For example, our experiments show that AFLFast probably does improve on AFL, though perhaps not as much as that paper made out (note we didn't do enough experiments ourselves to make that definitive statement).

It was a very interesting paper, thanks for writing it.

Now to figure out how to fuzz my program...

This paper is absolutely fascinating.

Fuzzers have some inherent nondeterminism, meaning that some runs agains the same program will differ in how many bugs they find.

It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous. We dont actually know if a lot of published results are, well, really an improvement.

There is also a fascinating bit towards the end. It turns out doing the dumb thing and not tracking code coverage finds more crashes faster than intelligent coverage tracking, but only with good seed inputs (see page 7).

The last comment is not justified by the paper. The paper shows one case where the dumb thing finds tons of crashes, but several other cases in the same figure where it finds zero crashes. And the figure there is just one target program. More experiments on this issue might be interesting.

One other thing to point out: Finding crashes and finding bugs are not the same thing, as section 7 carefully argues. It could well be that many of those crashes are duplicates. The blog post I linked above also summarizes this point.

I didn't initally understand this from your summary, so trying to rephrase it a bit to how I understand it after glancing over the paper:

The paper's thesis is that most (?) currently claimed advances in fuzzing algorithms were not proven correctly enough (a.k.a. "in a statically rigorous/significant way") to actually be advances. Meaning, they may well have been just random fluctuations. Then, the paper proposes how to correctly compare fuzzers, and what the community needs to be able to do that (e.g. an agreed upon common body of codebases to be fuzzed, a set of known fuzzable bugs in those codebases, minimum fuzzing time, etc.)

> It turns out pretty much every fuzzer evaluation to date has not been statistically rigorous.

Shall I tell you a not so well hidden secret? Almost nothing published in computer science, let alone in IT security, is statistically rigorous.

That's not true; plenty of papers in the systems field perform careful and rigorous experiments. This doesn't even include the theory side of stuff, where you only have proofs, and no statistics to get wrong.

Fuzzers don't have to be non-deterministic.

Choice bit from abstract: "We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer."

Note that some papers do come close to meeting what we view is the right standard. Table 1 shows that T-Fuzz does pretty well, for example.