Logically, a bug that prevents someone from doing something and a missing feature that prevents someone from doing something are equivalent, they shouldn't be thought of as two separate activities but should be evaluated via the same framework. The list of all defects in a piece of software is infinite so it becomes impossible to erase all defects. It's about prioritizing with finite resources.
> You can't predict the harm from a small bug when AI can chain complex exploits and when your software is used in mission critical applications as a data store.
Yes, any bug can theoretically be part of an exploit chain, just like any tumor detected can theoretically be the tumor that kills you. This is why the detection paradox exists. Once you become aware of the tumor, you face a duty to try and mitigate it. But we know from autopsies that people die with all sorts of tumors that didn't kill them.
Similarly, if we bothered to do autopsies of software products after sunsetting to discover a comprehensive list of CVEs, we'd discover a ton of bugs we were better off not knowing about because they ended up not making an impact but we would have diverted resources if we knew about them.
This is not an argument that we should remain totally ignorant of all bugs, but a refutation of the "common sense" belief that more knowledge is always better.
I guess it really depends on the severity of the defect. Would it allow some really significant exploit? -> Important to fix.
Does the defect only prohibit one very esoteric use-case which basically never comes up (and was thus never reported by a human user and only now that we have an infinitely patient machine checking every completely unlikely code path is it detected) -> more productive use of your time to develop a new feature which would be used by more people.
Firstly, the cost of fixing a bug once you know it exists is usually minimal compared to the cost of discovering that bug. This is the inverse of medicine, where the cost of testing (at least in terms of health outcomes for the patient) is typically minimal compared to the cost of treatment.
Secondly, if you have an asymptomatic illness, this does not typically make you more vulnerable (hence why it's asymptomatic). Like you say, you typically only die of one thing, so if you know the patient won't die of a particular diagnosis because something else is more likely to kill them, then it might not be worth making that diagnosis. But for software, you can be simultaneously vulnerable to lots of things at once, and any of these vulnerabilities can be the one to "kill" you. Some vulnerabilities may be easier to find and exploit (or do more damage) but in principle any active vulnerability can be exploited.
Thirdly, having multiple vulnerabilities makes software more vulnerable overall, because those vulnerabilities can potentially be chained together to create more serious exploits than each vulnerability would have represented on its own. To a certain extent this can be true in medicine as well, but generally it isn't.
Fourthly, we can't cure death (yet?) but we can cure all the vulnerabilities that we've found so far. There is no known upper bound on the lifespan of a piece of software, and we know that older code tends to have fewer bugs in that newer code, presumably because those bugs have been fixed over time. This is different from health issues, where a given condition may be chronic or terminal and therefore at a certain point there's nothing you can do to fix it.
Finally, ultimately in healthcare it's your body, and your decision was to what you do with it (assuming you can find a doctor willing to do what you want). If you have a condition and you want it cured, even if it is completely benign and will have no impact on your life whatsoever, then you can do that. (Again, assuming you have the resources and the condition is curable). But a software vulnerability does not affect you as the developer of the software, it instead affects your users/clients/customers. Therefore there are different dynamics in play. If I sell you a tool and I know it could be vulnerable, but I don't try and find out what those vulnerabilities are, then you're ultimately the one on the hook when all your data gets deleted or leaked.
Boy, I wish!
> Like you say, you typically only die of one thing, so if you know the patient won't die of a particular diagnosis because something else is more likely to kill them, then it might not be worth making that diagnosis.
You can't know someone didn't die of X until after they died, that's the entire point of this dilemma.
As to your second point, you don't know exactly what a person will die of, but you can make statically sound estimates. You generally won't test unlikely potential causes of death (either because the chance of a positive result is too small, or because you know there are other, more serious conditions). But once a vulnerability is known, it is infinitely reproducible, which means the chance of the vulnerability being "fatal" (assuming the conditions exist for it to be exploited) is essentially 1.
The other flaw in the analogy is that in medicine, the patient chooses to do the tests on themselves. But in software security, there is an adversarial attacker as well as the "doctor" trying to defend the software. And in general, if the defender can find a vulnerability, they must assume that an attacker could already have found the same vulnerability (because the attacker and defender have largely the same tools and information at their disposal, especially in open source contexts), in which case, as discussed previously, that vulnerability also needs to be assumed to be fatal.
The key thing you forgot, which makes this approach non-sensible, is the omniscience, without which you have no clue whether your posture towards a new bug-finding tool is good or not, since you don't know whether it will mostly find those bugs that are not worth fixing and the time wasted on fixing them is detrimental.
Similarly, how is ignorance better than informed decision when ignorance also includes missing critical security bugs?
This isn't an argument against all automated bug detection, just that it needs to exceed a threshold beyond just "It has found at least one real bug not found via other means" which is the seductive threshold people want it to be.
Go on - define a bar here.
> just that it needs to exceed a threshold
Same. Be specific.
That very much depends on got you define “bug” and how you specify your product requirements. Also, you have to assume the specs are correct.
But what applies to humans doesn't apply to software. First, to pursue the metaphor, codebase can definitely die/be compromised/crash because of multiple reasons at the same time; second, there is virtually no risk, out of man-hours, in fixing a bug.
It seems like this option is being taken away by LLMs for all major and widely used open-source projects.
For better and worse.
You would think so but medicine has faced this paradox since the development of modern testing technology.
You would naively think more testing is always better. More sophisticated practitioners then talk about Type I vs Type II errors and the problem of false positives.
But even if you were able to come up with tests that had 100% accuracy, there are times when you don't want to run that test on someone. The key distinction is that people only ever die of one thing and having a tumor that doesn't kill them is often better than risking the surgery to remove the tumor.
Similarly, there are some bugs you don't want to find since the harm from the people it affects is smaller than the opportunity cost of fixing it. Ignorance of the bug is better than knowing about it and having to make an affirmative decision not to fix it.