I've been really enjoying using Quickcheck & friends recently for test cases, which is essentially that, but doing the test case generation while the tests are running (and if it finds an error, trying to reduce that error to its simplest form). It's been very useful for catching little things that I'd forgotten about, like handling a particular case, or unusually shaped data, or things like that.

The biggest difficulty is trying to create good test cases. Unlike normal tests where you're asserting that some specific case is true, with quickcheck-style tests, you're instead trying to find an invariant in your code that will always be true. So it's quite easy to do "this algorithm never throws an error" but harder to do "this algorithm returns true in these cases and false in these cases". That said, a lot of tools in this space give you a lot of flexibility for generating cases, like saying "generate strings matching this regex" or "generate these arbitrary primitives and map then into the correct structure with this function".

The randomness involved isn't ideal, because you can't necessarily guarantee that the same errors will always show up every time, but it's a good habit to copy the failing generated case into its own test as a kind of regression test. And obviously still write a lot of the standard test cases for more obvious problem points.

Quickcheck tests are especially easy to use in cases where you wrote code that operates in two directions. For example, saving then loading a file should result in the same data, or a round-trip through a conversion process shouldn't modify the data. Alternatively, if there are two paths to do something, you can verify they are identical, for example saving a file and loading it compared to online syncing a file to a server, should result in the same file on the other end.

You might want to check out mutation testing. It's not the same at all obviously, but it has some advantages that it's a finite process (you know if you're done), and it's often fairly easy to write tests when you get a mutant. It's a much less cognitively demanding process while at the same time finding a lot of problems with your test suite.

I really like the idea of mutation testing, but I've never found a situation where I've got it to work well. Either the project just didn't have the test support needed to make it work, or the mutation toolkit wasn't mature enough, or it didn't work with the specific tools I was already using. When you suggested it, I had a go again with a project I'm working on using Vitest and Striker, but unfortunately Striker just doesn't work with Vitest yet.

I'd definitely love to use it more though, it's one of those things I come back to every so often, unfortunately thus far without much success.

There are two improvements that I think can be made to quick-check style testing. One is replacing the random source with bytes from a buffer and having a way to go from test cases to bytes. Hypothesis (python) does this. This means that you can connect a fuzzer like afl instead of generating purely random inputs. Another is setting up test-runners. If you have automated tests as part of CI, you want a fixed seed and deterministic ‘random’ inputs and a small number of runs so tests are fast and reliable. You can still catch obvious bugs but less likely bugs are harder to find. But if you can easily find all the tests and run them all the time with many more different inputs, hopefully you’ll have a much higher chance of finding rare bugs.

One problem though is that being better at finding bugs isn’t always great: if you find bugs you might feel the desire to fix them but for many software teams, having rare bugs is acceptable, even if they aren’t rare in absolute terms (ie a 1-in-10,000,000 when you have a billion chances for it to happen a day)