News for hackers. Stuff that matters.
Re:How deep are your unit tests? by tikhonj on Monday December 10, 2012 @04:36AM (#4898267)
This is one of the reasons I really like property-based testing à la QuickCheck[1]. The core idea is simple: you come up with an invariant for your code and the testing framework checks this invariant with randomly generated inputs. So when you're writing a test, you just have to come up with interesting invariants; you do not have to guess which inputs are edge cases.

[1]: http://www.haskell.org/haskellwiki/Introduction_to_QuickChec...

This may sound complicated and seem like overkill, but I've actually found it easier to use this style of tests for much of my code. Having randomly generated inputs can help find edge cases I did not even consider when writing the code in the first place--this addresses the bias issue you are worried about.

I also find that this style of test results in concise, easy to read tests. The usual "hello world" example uses the reverse function for lists. We want to ensure that for all lists, reversing it twice does nothing. The test would look like this:

    prop_reverseTwice ls = reverse (reverse ls) == ls
It's very easy to tell what invariant you're testing for! (The prop_* name is a convention for this sort of test.)

While real invariants are often more complex, I find they are still usually easy to read right from the code for the test.

Anyhow: you should really try QuickCheck or something like it for your language of choice.

Re:How deep are your unit tests? by danieldk on Monday December 10, 2012 @05:04AM (#4898337)
I agree that QuickCheck is really nice. It helped me spot some errors in e.g. tree diff'ing code that I wrote. QuickCheck especially works well for fundamental data structures and algorithms, where invariants are clear and simple.

However, in other cases, the usage is less obvious. For instance, suppose that you are writing, say a URL parser. What do you want QuickCheck to generate and check? If you generate too broadly, the invariant check becomes a parser itself, if you generate too specifically you might introduce bias again.

Re:How deep are your unit tests? by tikhonj on Monday December 10, 2012 @05:26AM (#4898401)
That's a good point. I remember having exactly that problem when writing tests for my toy language's parser. Generating random valid programs was actually tricky for several different reasons. In fact, I ultimately gave up on QuickCheck completely for that project, but mostly out of laziness.

Amusingly, I also wrote some tree diff code in Haskell recently. However, I did it rather poorly so I'm going to have to rewrite it soon (probably some time during winter break). I definitely agree that algorithms like this are particularly well-suited for QuickCheck.