> All determinism guarantees is that you'll either reliably detect a bug or reliably not detect a bug, and hence get a nice aesthetic line of green checkmarks followed by red crosses in your test history. Or just get a nice looking line of green and not even know the bug is there. This seems like a small thing to sacrifice to detect more bugs.

The situation you want to avoid is this:

Run tests, find bug.

Try and fix bug.

Rerun tests, tests pass.

Deploy fixes to production.

Later find out that the bug is not fixed, your second run of tests simply didn't hit the right combination.

We print the rand seed and allow tests to take a set seed. This allows us to use random generated data and still have deterministic results for reproducing failed cases.

Yep, this is a really useful approach, and I highly recommend anyone here who is thinking of trying this stuff out to take a note of the seed (or ensure it's printed) as the first time you realise you want to do this it's often too late!

I've used the same technique when running randomly generated traffic when testing telecom systems. Very useful to be able to re-run the same random sequence!

It's the second case that's more important: if you find a bug in production, and the test that covers that functionality is green, then there's a problem with your test suite.

Truth is, there's no need for an either/or holy war about this. The two styles are easily mixable: property based testing for broad coverage, specific runs for specific coverage.

A few good points have already been made in the other responses, and I want to add an additional observation: To me, the core of the issue is not in choosing between "non-deterministic" testing vs. a tiny set of hand-chosen examples.

Rather, to me, the issue is exhaustive testing vs. non-exhaustive testing. Typically, critical issues in complex systems only arise extremely rarely. Consequently, randomized testing typically does not bring them to light. There may definitely be cases where it is useful, and it can certainly easily be more exhaustive than a small hand-picked set of unit tests. But the core issue is still that I typically need exhaustive tests to be more certain about relevant properties.