News for hackers. Stuff that matters.
Re:Property-based testing in practice [pdf] by naasking on Tuesday April 02, 2024 @08:41AM (#39905067)
The problem with your hierarchy is that there's no empirical evidence supporting it. Small unit tests have not empirically been shown to have benefits over integration tests, and test-driven design has failed to show a benefit over tests written after the fact. The only thing that seems to matter is that tests are written, and the more tests, the more chances of finding a bug. That's it. So your list is actually:

* integration and unit tests: since these are manually written, they scale poorly but are simple.

* property tests: since these are semi-automatic they scale better but are a bit more complicated to setup.

* fuzzing: almost fully automatic, although I don't differentiate this much from property-based testing.

* mutation based testing

Re:Property-based testing in practice [pdf] by pfdietz on Tuesday April 02, 2024 @11:12AM (#39906634)
Is mutation based testing only better because it forces more tests to be written to kill the mutants?

Also, mutation based testing is really orthogonal to the others, since it's a way of evaluating the adequacy of the tests, not of actually testing. One could easily imagine using PBT/fuzzing to generate (and then simplify) tests with the express goal of killing mutants.

Re:Property-based testing in practice [pdf] by eru on Tuesday April 02, 2024 @11:16PM (#39913275)
> Small unit tests have not empirically been shown to have benefits over integration tests, [...]

Do you have any links to these studies?

> * property tests: since these are semi-automatic they scale better but are a bit more complicated to setup.

Once you are in the groove, I find property based tests to be simpler (or not harder) than example based tests. But that's when I am writing tests as I am developing the system, ie I take testability into account in the design.