News for hackers. Stuff that matters.
Re:Java Mutation Testing by the_af on Thursday May 07, 2015 @06:36PM (#9508840)
Your tests should be written in a way that they test there is a consistent ordering, if that's a relevant property for you. If they don't, you have a problem. If they do, I guess the mutation framework will merely point out your tests don't detect some changes; it doesn't necessarily mean they are bad tests, it's merely a "smell" that deserves further investigation.

(By the way, I think you're overstating your case when you claim "there are plenty of places" where it doesn't matter. I think in the vast majority of cases, if you change a "<" to a ">", you introduce a bug!)

Re:Java Mutation Testing by TheLoneWolfling on Thursday May 07, 2015 @07:13PM (#9509032)
I agree that in the majority of cases it will cause bugs. But what I'm saying is that there are cases where it can legitimately not cause a bug.

The problem I have is this: this testing framework explicitly states the following:

> The quality of your tests can be gauged from the percentage of mutations killed

So, you write your tests to test that there is a consistent ordering, it changes the "<" to a ">", finds that all tests still pass (because, as you said, you're only testing if there is a consistent ordering, because that's the only thing that matters in this case), and considers your tests lower-quality because of that.

This is a flaw most mutation testing - and, for that matter, test driven design - suffers.

For another flaw: there are many things that end up along the lines of a bunch of shortcut "fast paths" at the start, with a slow path at the end that does slow but exhaustive checking.

For example, if you're checking that two objects are equal you may insert a check at the start to see if they are the same pointer, and if so return `true`. If you're doing division in software you may check if you're dividing by 1 instead of doing the computation. That sort of thing.

Any mutation that introduces false negatives into the shortcuts (but no false positives) won't be flagged by mutation testing.