When I'm writing tests, I'm not so concerned about what the tool tells me about which lines are covered, I like to work through the mental exercise of knowing which basis paths are covered. If a function has too many for me to reason about, that is a problem in itself.
As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. If you have many methods in a class with only one basis path, what purpose does the class serve. These testing concerns may be the code smell that points to deeper problems
One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way.
Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), people split it up into multiple classes, and then test each bit. This allows them to use the class architecture to mock things, but really has just multiplied out the amount of code. And in the end, you're running tests over the golden path probably even less. It's even possible to have 100% of the code covered, and not run the golden path, because you're always mocking out at least one bit.
Thus it is often a test of the implementation's assumptions when first written, and even worse, when the code is maintained/edited, the test is merely changed to get it to pass, because unit tests with mocks are usually:
1) fragile to implementation 2) opaque as to intent
Whereas input/output integration points are more reliable, transparent, and less fragile to implementation changes if the interface is maintained.
However, if you must do mock-level interaction testing, Spock has made it almost palatable in Javaland.
This is one area where functional fans get to make the imperative folks eat their lunch.
printf("hello world");
And the test is: mock_printf(string) {
if string != "hello world" then fail;
}
Which is basically just duplicating your code as tests.Usually because interfaces need to be clearly/well defined and overridden... In JS there are tools (rewire, proxyquire, etc) that can be used with tests in order to easily inject/replace dependencies without having to write code to support DI nearly as much. In fact, I'd say that there's usually no reason not to be very close to 100% coverage in JS projects.
On .NET it is a plain library, which requires the VS Ultimate editions to be useful and on Java it was mostly third party libraries.
C# design team is considering adding proper support as language feature, but it seems to be a very far away feature still.
C++20 might get contracts, but C++17 just got ratified, so who knows.
D, Ada 2012 and SPARK do already support contracts.
http://dlang.org/spec/unittest.html
It's been very effective at improving the overall quality of the code. And because of CTFE (Compile Time Function Execution) and static assert, many code correctness tests can even be run at compile time.
I wonder what that would look like...
We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger tools should be able to hone in on areas where the result diverges from expectations.
We can achieve this in dependently-typed languages like Idris. First we define a datatype 'Equal x y', which will represent the proposition that expression 'x' is equal to expression 'y':
data Equal x y where
Refl : (x : _) -> Equal x x
There are two things to note:- There is only one way to construct a value of type `Equal x y`, which is to use the constructor we've called `Refl`.
- `Refl` only takes one argument, `x` (of arbitrary type `_`), and it only constructs values of type `Equal x x`. This is called "reflexivity", thus the name `Refl`.
Hence if we use the type `Expr x y` anywhere in our program, there is only one possible value we can provide that might typecheck (`Refl x`), and that will only typecheck if `Expr x x` unifies with `Expr x y`, which will only happen if `x` and `y` are the same thing; i.e. if they are equal. Thus the name `Equal`.
This `Equal` type comes in the standard library of all dependently typed languages, and is widely used. To use it for testing we just need to write a test, e.g. `myTest`, which returns some value indicating pass/fail, e.g. a `Boolean`. Then we can add the following to our program:
myTestPasses : Equal myTest True
myTestPasses = Refl myTest
This will only type-check if `Equal myTest myTest` (the type of `Refl myTest`) unifies with `Equal myTest True`, which will only be the case if `myTest` evaluates to `True`.[0] https://developer.salesforce.com/docs/atlas.en-us.apexcode.m...
And another example of something I'd want checked by the language is exception throwing / handling. It's another one of those places where code coverage won't help you unless you already know what you're looking for. Languages are getting better about it, but in general, handling errors is hard.
In haskell you embed domain specific languages when you require side effects:
class MonadState s m where
get :: m s
put :: s -> m ()
This is like an interface specifying the basics of all stateful computations.You can use different implementations for production and testing without changing any code so mocking is built into everything.
It ties together related algorithms. You want "addDays(n)", "addHours(n)", "addMinutes(n)" and so on to be in the same class, even if they're one-liners.
I note that the examples you give have trivial test cases - presuming you don't care about overflow, and if you do, then those methods now have basis paths, whether explicit or implicit depends on the language.
It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim.
The other option is to accept that perfect coverage is not feasible, or possibly even a trap, and that you just need to cover the interesting bits.
The halting problem is an irrelevant, Ivory-tower distraction in production code.
That's the idea at least. All focus on high coverage, whether line coverage or mutation testing, creates an incentive to remove redundant robustness checks, and maybe that's not such a good idea after all. But that's a problem with all unit testing, and not just with mutation testing.
If this change doesn't cause a test to fail, you're not testing edge cases for this comparison.
(Of course that's assuming you want perfect coverage.)
while ( j < JMAX) {
i = callback(j);
if( i >= FINAL) break;
...
If, in the specific circumstances of this use, callback(j) will always equal FINAL before it exceeds it, a test for equality here will not cause the program to behave differently.The fallacy of your argument is that == is not an assertion that its arguments should always be equal; it is a test of whether they are at some specific point in the algorithm.
"Pex uses a constraint solver to produce new test inputs which exercise different program behavior. "
So this is not lines, but I guess.. branches and numerical limits. Either way it's cool and I wish it was integrated somewhere. They've got a online demo thing http://www.pexforfun.com/
[1]https://www.microsoft.com/en-us/research/publication/pex-whi...
[0] https://blogs.msdn.microsoft.com/visualstudio/2015/09/30/int...
However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-every-commit test suite just exercises with a selection of values likely to expose obvious problems.
The hard part is the percentage rgb() values, of which there are technically an uncountably infinite number (since any real number in the range 0-100 is a legal percentage value). For those I generate all 16,777,216 integer values, and verify that converting to percentage and back yields the original value.
https://github.com/ubernostrum/webcolors/blob/master/tests/f...
The fun part is people who criticize the generation of the integer triplets; yes, it's three nested loops and that's bad, but the total number of iterations will always be 16,777,216 no matter what algorithm you decide to use to generate them. So it uses nested loops since that's the most readable way to do it.
You're also making it sound like somebody promised you that tests can prove the absence of bugs, when that was never the bargain and smart people have already told you so, probably before you were born even.
> Testing shows the presence, not the absence of bugs
Dijkstra (1969)
Definitely not a tragedy.
I think when you take all of the time wasted on useless tests written merely for the sake of having tests, that waste is tragic. You could be doing anything else with that time.
It seems to me that waste is unavoidable with teams newer to automated testing and may simply be part of the cost of using automated tests. If that is the case then it seems better to compare the cost of bugs with no automated to cost of superfluous tests. In that comparison extra tests definitely seems like the lesser of two evils, even without hard numbers. I would prefer hard numbers because my intuition could be wrong.
1. Take the number of hours spent writing tests.
2. Multiply by whatever percentage of the tests are unnecessary.
3. Multiply by the labor cost per hour. Or revenue that was not made (e.g. if you could have billed those hours to a customer, but didn't).
The resulting number could either be a big deal or not, depending on how big your organization is and how much time you spent on superfluous tests.
Automated tests can do so much more than manual tests that shops still living by manual tests either have a damn good reason or are just as wrong as people who argued against Revision Control Systems (or people who argued for gotos instead of functions).
Automated tests will be executed identically each time, so no missing test cases because someone slacked off or made a typo. Automated test can serve as examples for how to use the code. Automated tests can aid in porting to new platforms, once it builds you can find all the bugs your care about swiftly. Automated test can be integrated with documentation, tools like doxyegn and mdbook make this easier.
Automated tests enable Continuous Integration. Are you familiar with Travis CI or Jenkins? If not, imagine a computer that a team commits their code to, instead of directly to the mainline Revision Control (Git master, svn head, etc...). That computer builds the software, runs all the tests, perhaps on every supported platform or in an environment very close to production, then only merges commits that appear to fully work. This doesn't completely eliminate bugs and broken builds, but the change is so large that teams without it are at a clear competitive disadvantage.
When integrated into process tests can be used to protect code from changes. If a test exercises an API and the team knows that is the purpose then when they change things in a way that break the test they shouldn't... This sounds vague or obvious, but consider this: At Facebook and Google they have a rule that if it is not tested new code doesn't have to care if it breaks. Both companies have team that make broad Sweeping changes. Facebook wrote a new std::string and Google use clangtools to make automated changes in thousands of places at once. Even if code breaks or APIs change as long as tests pass these people can be sure that they negatively impacted the product and are following their team's rules.
Automated Tests can... This list could go on for a very long time.
I think it should be a mix. I disagree with the "extreme minority" portion for most projects. There are times where a manual test is the right answer and there's exploratory manual testing.
Obviously though running through a long sequence of testing manually for every code change is crazy. And some sort of CI setup like you describe is a must for every project this day and age.
Then there's also the question of unit tests vs. system/integration tests...
Why not prove the code is correct instead? Should be much cheaper than 100% coverage and more certain.
Mutation testing systems normal use fairly stable operators (e.g changing a > to a >=). In most locations in the code changes such as these will have only a subtle effect.
On the other hand, if you do cover all your inputs, you've covered all the cases regardless of what % code coverage or path coverage you have.
To prove it works with all possible inputs, there are other tools at your disposal.
Brute force.
Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results.
Me I have a pitiful amount of unit tests for the main codebase I work on. One module which does have unit tests, also had a bug that resulted in us having to replace a couple of thousand units in the field.
Otherwise most of the bugs that chap my keister aren't about procedural correctness, they're about state.
A + B == B + A
A + 0 == A
(A + B) + C == A + (B + C)
A + B != A + C (given B != C)When you have one available, testing against the results against a reference implementation is a little more direct. Though, given how easy those property tests are, you might as well do both.
Some times I have, for example, matlab code that I 'know' is right and want to verify that my C/Python/Julia... code does the same thing. So then I just call the matlab code, call the new code and check if they're the same.
Or I have a slow exact method and want to see if my faster approximation always gives an answer 'close enough' to the correct answer.
This sounds pretty sketchy at first, but it turns out that almost all of the difficulty (and hence almost all of the bugs) are in getting the last few bits right, and the last few bits of a much-higher precision result don't matter when you're testing a lower-precision implementation, so combined with some basic sanity checks this is actually a very reasonable testing strategy (and widely used in industry).
FLOAT_EQ(sqrt(x) * sqrt(x), x) && sqrt(x)>=0.0
is a pretty good test for the square root even though it doesn't compute it.I'd like to hear about real world test scenarios where all possible inputs are tested.
A sane type system would be a start for example. And it's no coincidence that functional programming and practices with emphasis on immutability are on the rise; Rusts ownership system is a direct consequence as well.
TDD _is_ important, if simply for enabling well-factored code and somewhat guarding against regression bugs. But - decades after Dijkstras statement (which someone has already posted in this thread) - the code coverage honeymoon finally seems to be over.
The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches.
It is quite useful mechanism for checking inputs, formats, behaviour patterns (if you have two solutions but one model, one simple that works 100% but is slowish and one more complex but very fast).
See: https://github.com/dvyukov/go-fuzz#trophies and http://lcamtuf.coredump.cx/afl/
What I meant was what @AstralStorm said about mathematical and logic proofs that it works for all defined values.
Proofs are powerful, but you can still make errors in the proof or the transcription into code. And of course you don't know what other emergent behaviours will surprise you when the proved code interacts with unproved code.
Depending on your means and needs, you want to try both.
Just because you ran a function or ran a line doesn't mean it will work for the range of inputs you are allowing. If your function that you are running coverage on calls into the OS or a dependency, you also have to be ready for whatever that might return.
Therefore you can't tell if your code is right just by having run it. Worse, you might be lulled into a false sense of security by saying it works because that line is "covered by testing".
The real answer is to be smart, pick the right kind of testing at the right level to get the most bang for your buck. Unit test your complex logic. Stress test your locking, threading, perf, and io. Integration test your services.