News for hackers. Stuff that matters.
Re:Stranger Strings: An exploitable flaw in SQLite by adrianN on Tuesday October 25, 2022 @10:37AM (#33331021)
It is absolutely not easy to achieve the kind of coverage that sqlite has. You need massive amounts of tests for that. Of course coverage is not sufficient, you can exercise code without checking any properties at all. But to say that it gives a false sense of security is too harsh.
Re:Stranger Strings: An exploitable flaw in SQLite by capableweb on Tuesday October 25, 2022 @10:52AM (#33331273)
> It is absolutely not easy to achieve the kind of coverage that sqlite has.

It is though. Give me one project and I'll send you a test suite that "tests" 100% of executable lines. It won't have any actual assertions, but that's not what code coverage is about anyways, so hopefully you won't mind.

Number of assertions per line of code is about as good metric for how well tested something is as code coverage, which means just about nil.

Re:Stranger Strings: An exploitable flaw in SQLite by chasil on Tuesday October 25, 2022 @11:23AM (#33331832)
Compliance with the DO-178B standard was not easy for the SQLite maintainers. It required several years to achieve.

By adhering to the DO-178B standard, the testing is confirmed extensive and satisfies requirements for use in avionics.

"Rockwell Collins... introduced me to this concept of DO-178B. It’s a quality standard for safety-critical aviation products, and unlike so many quality standards, this one’s actually readable... one of the key things that they push is, they want 100% MCDC test coverage.

"Your tests have to cause each branch operation in the resulting binary code to be taken and to fall through at least once.

"At the machine code level... actually, MCDC’s a little stricter than that. Let’s not dwell on the details, but I had this idea, I’m going to write tests to bring SQLite up to the quality of 100% MCDC, and that took a year of 60 hour weeks. That was hard, hard work. I was putting in 12 hour days every single day. I was just getting so tired of this because with this sort of thing, it’s the old joke of, you get 95% of the functionality with the first 95% of your budget, and the last 5% on the second 95% of your budget. It’s kind of the same thing. It’s pretty easy to get up to 90 or 95% test coverage. Getting that last 5% is really, really hard and it took about a year for me to get there, but once we got to that point, we stopped getting bug reports from Android."

https://corecursive.com/066-sqlite-with-richard-hipp/

Indeed, SQLite is used for flight systems on the Airbus A350. No other major database vendor is certified to do so.

https://www.sqlite.org/famous.html

Re:Stranger Strings: An exploitable flaw in SQLite by Someone on Tuesday October 25, 2022 @11:08AM (#33331594)
> Give me one project and I'll send you a test suite that "tests" 100% of executable lines.

I don’t think you can do that with every project. For example, a function f may do:

  if (p == null) {
    abort();
  }
If all callers also do that test, there’s no way to hit that abort() line in the program. Your test suite may not be able to hit it either. Maybe f is a private function, or that test occurs on one of its intermediate values, or f accidentally has two identical tests:

  if (p == null) {
    abort(“p is null”);
  }
  if (p == null) { // copy-paste bug
    abort(“q is null”);
  }
  if (r == null) {
    abort(“r is null”);
  }
Even if it is possible to hit that line, finding a path through the code that passes in a null value can be extremely hard.

As a simple contrived counterexample, try getting 100% coverage on a function that does if(secure_hash(foo) == 0) bar();

Re:Stranger Strings: An exploitable flaw in SQLite by phoe-krk on Tuesday October 25, 2022 @11:35AM (#33332006)
> Your test suite may not be able to hit it either.

Minor nitpick, process terminations are unit-testable in general; see e.g. https://github.com/google/googletest/blob/main/docs/advanced...

Re:Stranger Strings: An exploitable flaw in SQLite by garaetjjte on Tuesday October 25, 2022 @12:26PM (#33332749)
They don't count assertions in testing coverage: https://www.sqlite.org/testing.html#coverage_testing_of_defe...
Re:Stranger Strings: An exploitable flaw in SQLite by jonhohle on Tuesday October 25, 2022 @12:32PM (#33332840)
What is fair in testing and what is being tested? If I can stub `secure_hash` and `bar`, that’s a pretty easy test to write. Assuming the symbols are visible at test time, stubbing those is relatively straight forward in C, Java, Bash, and Ruby, JavaScript, etc.

Even in the above examples, a linter should point out that the second `p == null` is inaccessible. I’m very pro test coverage, but static analysis shouldn’t be thrown out just because unit tests are available.

Re:Stranger Strings: An exploitable flaw in SQLite by adrianN on Tuesday October 25, 2022 @11:20AM (#33331784)
How much do you charge? Because adding assertions to tests that give MC/DC coverage is a lot cheaper than writing them. I know a couple of big companies who might be interested.