If the acceptance criteria is “would prevent every single past instance and every imaginable future instance”, then yes, no mitigation is every sufficient to address any problem in the world, so we might as well give up.
But I don’t think that’s the right lens to use.
You're in control of how much danger of accident you expose yourself to.
Nobody is in control of how much danger we are exposed to from other people who are actively trying to do us harm, who will keep going until they get what they're after or are stopped.
For most people, seatbelts are the former. Yeah, not perfect, but they reduce risk. For the latter, if you're known to be a seatbelt wearer, the attacker just does something where seatbelts don't matter.
Every new AI model introduces new capabilities and competencies, so we're not even sure what the true risk levels are yet for self-exposure in this category. The restrictions on AI may be like seatbelts and speed limits, or they may be like "if you install a 1000 HP turbojet engine in your Honda Civic it will no longer be road legal". And this analogy also includes how the first cars had speed limits set low enough to not risk the horse industry, i.e. we may be too cautious.
But I think people misinterpret the principle to mean that only perfect solutions have any value.
When in reality defense in depth is the opposite principle: you scan incoming emails for phishing, and that’s good but imperfect. You do mandatory training, and that’s good but imperfect. You use RBAC to limit blast radius, and that’s good but imperfect. And so on.
Among tech people, especially on HN for some reason, there’s this odd thinking style that goes: 1) company announces security measure, 2) the measure could possibly limit my freedom to do whatever I want with the company’s products, 3) I don’t like that, 4) I can come up with scenarios where the security measure is not sufficient, entirely on its own, to address the claimed risk, 5) therefore the security measure does no good at all, 6) therefore this is a PR smokescreen to disguise their desire to capriciously fuck with me out of pure malice, and I am angry about it
As with clever, careful serial killers, it's tough to count the ones we haven't caught.
It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere, but the point is that in terms of actual attempts, we've seen a single one and it wasn't even successful despite years of prep.
No, we can't, as that happens a lot via non-serial killers.
A truly successful serial killer is likely one who hides in that noise. No taunting the cops, distributed geographic locations, random methods, avoiding calling cards, and careful not to leave too many traces.
It seems likely that some of the 350k unsolved homicides in the US can be explained this way.
> It's possible there are infiltrators who are still working on long-term infiltration and haven't yet attempted to add any malicious code anywhere…
Or the code's already there, latent, as it would've been in the XZ case, which got discovered by chance and someone very dedicated to looking into a performance glitch.
Since we do not know the ratio to undiscovered this "1-2" is meaningless to assess the risk of this sort of attack.
> A subsequent investigation found that the campaign to insert the backdoor into the XZ Utils project was a culmination of over two years of effort, starting in 2021, by a user going by the name "Jia Tan". They used sock puppetry in a pressure campaign against the original maintainer of XZ Utils, eventually being given maintainer permissions on the project.