By adhering to the DO-178B standard, the testing is confirmed extensive and satisfies requirements for use in avionics.
"Rockwell Collins... introduced me to this concept of DO-178B. It’s a quality standard for safety-critical aviation products, and unlike so many quality standards, this one’s actually readable... one of the key things that they push is, they want 100% MCDC test coverage.
"Your tests have to cause each branch operation in the resulting binary code to be taken and to fall through at least once.
"At the machine code level... actually, MCDC’s a little stricter than that. Let’s not dwell on the details, but I had this idea, I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work. I was putting in 12 hour days every single day. I was just getting so tired of this because with this sort of thing, it’s the old joke of, you get 95% of the functionality with the first 95% of your budget, and the last 5% on the second 95% of your budget. It’s kind of the same thing. It’s pretty easy to get up to 90 or 95% test coverage. Getting that last 5% is really, really hard and it took about a year for me to get there, but once we got to that point, we stopped getting bug reports from Android."
https://corecursive.com/066-sqlite-with-richard-hipp/
Indeed, SQLite is used for flight systems on the Airbus A350. No other major database vendor is certified to do so.
I don’t think you can do that with every project. For example, a function f may do:
if (p == null) {
abort();
}
If all callers also do that test, there’s no way to hit that abort() line in the program. Your test suite may not be able to hit it either. Maybe f is a private function, or that test occurs on one of its intermediate values, or f accidentally has two identical tests: if (p == null) {
abort(“p is null”);
}
if (p == null) { // copy-paste bug
abort(“q is null”);
}
if (r == null) {
abort(“r is null”);
}
Even if it is possible to hit that line, finding a path through the code that passes in a null value can be extremely hard.As a simple contrived counterexample, try getting 100% coverage on a function that does if(secure_hash(foo) == 0) bar();
Minor nitpick, process terminations are unit-testable in general; see e.g. https://github.com/google/googletest/blob/main/docs/advanced...
Even in the above examples, a linter should point out that the second `p == null` is inaccessible. I’m very pro test coverage, but static analysis shouldn’t be thrown out just because unit tests are available.
It is though. Give me one project and I'll send you a test suite that "tests" 100% of executable lines. It won't have any actual assertions, but that's not what code coverage is about anyways, so hopefully you won't mind.
Number of assertions per line of code is about as good metric for how well tested something is as code coverage, which means just about nil.