Re:Using Markov Chains to Generate Test Input
by kurtosis on Tuesday September 15, 2009 @04:35PM
(#824689)
Okay I understand where you're coming from. I'm really curious though, what were some of the bugs that showed up? Were they easy to fix or did they require adding a lot of special cases and patchwork to the code?
Re:Using Markov Chains to Generate Test Input
by emelski on Tuesday September 15, 2009 @05:28PM
(#824802)
There were about a dozen issues uncovered using this technique. I would say the majority of them were what I would call "real" bugs, as opposed to "sure, that doesn't work but nobody cares". One example is that gmake allows $$ in variable names, as in "FOO$$BAR=abc"; at the time our parser did not handle this correctly. I did not have to create special cases to fix these bugs, at least not any more so than any other parser feature.
There are a handful projects that don't follow this pattern, such as Mozilla, but once you've built those they cease to be useful for flushing out new bugs.
So, what do you do once you've built all the open source projects you can find, and you are still finding incompatibilities in your parser when you go to customers? Markov chain-based makefiles are a reasonable way to extend the breadth of our testing. Obviously it's not the only thing we do, but it is another valuable tool to add to the toolbox.