News for hackers. Stuff that matters.
Re:Rethinking software testing: perspectives from the world ... by pjc50 on Sunday July 11, 2021 @03:44AM (#27799111)
> The aim should always be complete coverage.

The problem with this is the requirement to build a completely controllable test fixture including OS syscalls. In practice that's very difficult.

How many people test their javascript in a sandbox that allows for arbitrary fault injection of calls into the browser API or DOM?

> If not, it will simply keep on searching, potentially for months, years, ... You need to be able to start the search for mistakes and keep it running for as long as you want.

You also need to ship, and in many startup environments you don't even have a deadline, it's just "as soon as possible".

Fuzzing is very effective at finding security issues. It does seem reasonable to try to design the security boundary of an application in a way amenable to fuzzing.

Re:Rethinking software testing: perspectives from the world ... by triska on Sunday July 11, 2021 @05:47AM (#27799596)
To clarify: Tests as I see them and described them above do not need to stop when the software is shipped. They are meant to run indefinitely and can continue also long after the software is shipped, and independent of the environment in which it was developed. For example, I found several mistakes in released versions of GNU Emacs with test cases that repeatedly do the same thing, over and over, until they reveal a crash or memory leak. A few examples of this approach:

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=496

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=576

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=726

Fuzzing is a useful approach and does reveal many issues with software. As I see it, the fact that fuzzing already tends to reveal mistakes in software is a good motivation to strive for methods that yield greater coverage based on systematic exhaustive search, so that also mistakes that occur in very rare situations are reliably detected.