Re:Controversial Programming Opinions (2012)
by josephg on Friday December 04, 2020 @04:03PM
(#25307945)
I use fuzzers extensively in software I write. I find it quite enjoyably humbling how quickly a newly minted fuzzer finds bugs.
I use a seeded random number generator. During development I seed it randomly. When crashes happen I output both the seed and iteration so I can hardcode that seed and reproduce the problem. And then in the CI I’ll often fix the seed so the test suite is stable.
But it’s also good practice to run fuzzers outside the context of a CI. You want your CI tests to be fast and stable. Fuzzer tests can also be slow and thorough. Foundationdb has dedicated hardware that runs their fuzzer 24/7 and spits out any errors it finds. If that fuzzer finds problems it doesn’t break the build. But someone will take a look, reproduce the problem, hopefully convert the bug into a unit test and fix the issue.
I use a seeded random number generator. During development I seed it randomly. When crashes happen I output both the seed and iteration so I can hardcode that seed and reproduce the problem. And then in the CI I’ll often fix the seed so the test suite is stable.
But it’s also good practice to run fuzzers outside the context of a CI. You want your CI tests to be fast and stable. Fuzzer tests can also be slow and thorough. Foundationdb has dedicated hardware that runs their fuzzer 24/7 and spits out any errors it finds. If that fuzzer finds problems it doesn’t break the build. But someone will take a look, reproduce the problem, hopefully convert the bug into a unit test and fix the issue.