Re:Rethinking software testing: perspectives from the world ...
by jka on Monday July 12, 2021 @07:12PM
(#27816190)
Improving coverage is a good goal, although (depending on the codebase) I think it can provide a false sense of robustness.
Generally more coverage is good, yes; but there can be an element of risk around "testing two inputs in isolation", to borrow terminology from the article.
You may have coverage for every line of function A and every line of function B, but that doesn't guarantee that results of the two - when combined dynamically at runtime - are going behave the way that is expected and intended.
I like your idea about tests that are continuously searching. In theory I also think it'd be possible to have tests that run on a codebase near-immediately as a developer types (even between commits); ideally exercising the data flow paths that are most directly affected by each syntax tree change first, and expanding out from there.
Generally more coverage is good, yes; but there can be an element of risk around "testing two inputs in isolation", to borrow terminology from the article.
You may have coverage for every line of function A and every line of function B, but that doesn't guarantee that results of the two - when combined dynamically at runtime - are going behave the way that is expected and intended.
I like your idea about tests that are continuously searching. In theory I also think it'd be possible to have tests that run on a codebase near-immediately as a developer types (even between commits); ideally exercising the data flow paths that are most directly affected by each syntax tree change first, and expanding out from there.