News for hackers. Stuff that matters.
Re:Stranger Strings: An exploitable flaw in SQLite by RHSeeger on Tuesday October 25, 2022 @09:35AM (#33330266)
Indeed. Tests do not tell you code is correct. They only tell you a list of ways it is not incorrect.
Re:Stranger Strings: An exploitable flaw in SQLite by aa-jv on Tuesday October 25, 2022 @09:38AM (#33330290)
.. and combined with coverage they also tell you what you are yet to test properly.
Re:Stranger Strings: An exploitable flaw in SQLite by fulafel on Tuesday October 25, 2022 @09:40AM (#33330318)
s/properly//
Re:Stranger Strings: An exploitable flaw in SQLite by kevincox on Tuesday October 25, 2022 @09:58AM (#33330525)
Actually: It tells you some things that are not tested properly.

Just because something has coverage doesn't mean that it is tested properly as evidenced by this bug even though sqlite has 100% branch coverage.

Re:Stranger Strings: An exploitable flaw in SQLite by Macha on Tuesday October 25, 2022 @10:34AM (#33330990)
Yeah, coverage (or rather 100% - coverage) is basically a lower bound on what's not tested.
Re:Stranger Strings: An exploitable flaw in SQLite by masklinn on Tuesday October 25, 2022 @10:51AM (#33331257)
Even "coverage" is not enough, what 100% coverage? Lines? Branches? Paths?
Re:Stranger Strings: An exploitable flaw in SQLite by Macha on Tuesday October 25, 2022 @01:40PM (#33333764)
That's my point, coverage (whatever your chosen metric) can only prove the absence of tests, not their presence
Re:Stranger Strings: An exploitable flaw in SQLite by PathOfEclipse on Tuesday October 25, 2022 @12:01PM (#33332408)
I think this thinking is short-sighted. Tests increase your confidence that code is correct, and they are much cheaper to do that formal verification, which itself still requires a leap of faith that there is no bug in the formal verification system, its implementation in the compiler toolchain, or the assumptions made about the underlying hardware.

You can cast doubt on anything. Will you be alive 30 seconds from now? You don't know for sure! But you can be pretty confident. Similarly, testing can increase confidence in code correctness. SQLite has an excellent security history overall, and it's testing strategy is part of the reason for that success.

Re:Stranger Strings: An exploitable flaw in SQLite by pclmulqdq on Tuesday October 25, 2022 @01:13PM (#33333394)
Formal verification also becomes practically impossible at a certain level of complexity.

SQLite is better tested than many silicon chips, so it's hard to fault their testing strategy.

Perhaps a few static analyzers should be run on the codebase to find possible bugs like this.

Re:Stranger Strings: An exploitable flaw in SQLite by hulitu on Tuesday October 25, 2022 @04:04PM (#33335454)
> Formal verification also becomes practically impossible at a certain level of complexity.

Then reduce complexity.

Re:Stranger Strings: An exploitable flaw in SQLite by function_seven on Tuesday October 25, 2022 @04:34PM (#33335777)
You can make things as simple as possible, but no simpler than that.

SQLite is a complex program, inherently. It's designed to allow users to do complex things, and as such cannot be simpler than its purpose allows.

I'm no formal verification expert—so I don't know if SQLite is above or below that fuzzy threshold—but "reduce complexity" is not always an available tool.

Re:Stranger Strings: An exploitable flaw in SQLite by nottorp on Tuesday October 25, 2022 @04:21PM (#33335627)
Well, int main (void) { return 1; } is most likely bug free. But it's not useful (except as /bin/true).
Re:Stranger Strings: An exploitable flaw in SQLite by jwilk on Tuesday October 25, 2022 @06:04PM (#33336743)
That would be a very buggy /bin/true implementation.
Re:Stranger Strings: An exploitable flaw in SQLite by nottorp on Wednesday October 26, 2022 @05:09AM (#33341350)
Right, but a correct /bin/false :)
Re:Stranger Strings: An exploitable flaw in SQLite by hulitu on Tuesday October 25, 2022 @03:39PM (#33335177)
> Tests increase your confidence that code is correct, and they are much cheaper to do that formal verification,

Your confidence as SW engineer. The purpose of a test is to look for unexpected behaviour, not for expected behaviour.

Re:Stranger Strings: An exploitable flaw in SQLite by RHSeeger on Wednesday October 26, 2022 @01:21PM (#33346709)
And to be specific, they generally look for _known_ unexpected behavior; in that you need to have a test for it.

Though clearly there's still unknowns, like when you're throwing random data at an api to see if you can find bugs. But even still, you know what _ways_ you're doing that. Tests generally won't turn up things nobody has thought to test.