I disagree that something like this would be a good thing though. Whether it's a power outage, financial crisis, or something else people will die. If that's what happens and society ends up adjusting course after then that's what happens, but I can't call people dying a good thing.
I don't think that's a good thing. But if we DON'T do anything, then even more poeple are likely to suffer. So while people dying is not a good thing, a thousand people dying is better than a million people dying because we don't do anything.
Same thing about burning your hand. It's not a good thing to burn your hand, but burning your hand once when you're young probably saves you from a more serious accident later on.
We both don't know all the possible futures and we don't know if a power outage or similar would kill 1,000 or 1,000,000. We also don't know if that loss would lead to the future we want.
This sort of thing was going to happen, with or without AI (state sponsored hackers, the US or China taking down systems they control, or whatever).
The later it happens, the more we will be down the path that lets it happen, and the worse the consequences. We are getting gradually worse consequences: AWS outages taking down websites, GCP outages taking down banks, attacks on fridges disrupting good supplies...
Its better for us to change path when something happens that causes dozens of people die rather than wait until its millions.
The OP didn't say we should make it happen so I'm not intending to put those words in their mouth. Its a small leap from "it might be a good thing if the power is out for days" to someone hearing or thinking that themselves and acting on it.
If it turns out we have a power outage, people die, and we learn a lesson and adjust course then so be it. That would be what happens in that future but it isn't necessary and it isn't a good thing even if today we think it leads to the best outcome - we can't know that without knowing all the outcomes and would be generalizing into only the ones we can imagine.
Blockchain despite all its flaws, is the canary in the coal mine. They have been dealing with an increasing amount of cyberattacks. Because it all happens publicly in the blockchain, there is no way to hide it.
However major bank gets hacked in the tune of over 100 millions and we get to know it happened because somebody who has no relationship to the hack turned his eyes to a bunch of suspicious transactions happening on Ethereum.
This is why I think AI is so stupid. It's just too powerful for anyone to control, especially with the inertial mass of capitalistic society powering it through the incremental, short-term advantages it gives.
Nuclear weapons are indiscriminate and make everyone worse off if used. And everybody has them. Your optimism is unfounded.
The difference is most people have not experienced any immediate effects of nuclear weapons. But people will experience the immediate effects of AI taking down the electricity grid, for instance.
If we're at the point where all that matters is scale and reinforcement training then the public facing aspects could entirely disappear and progress would still continue especially when given an infinite budget of taxpayer dollars.
Maybe the long term changes such blackouts would cause would be "worth it" in a utilitarian sense, but let's not hope for such damage.
And I'd go one step further to say that it probably can not be true. This has always been true and probably will always be true. All LLMs are doing is demonstrating that. And the LLMs of tomorrow will demonstrate that the most LLM hardened system of today is still just a vulnerable mess. So the conclusion is simple - things should not be connected to the internet unless the worst case scenario from hacking/exploiting is relatively low. If somebody wants to connect their coffee maker to the internet, for one can only imagine whatever reason, then the worst case scenario is a broken coffee pot - who cares, go for it. But connecting e.g. power, water, or other core infrastructure to the internet? No, never.
Just because something hasn't been proven to be a problem today does not mean it won't be a problem tomorrow. We shouldn't learned that yesterday, hopefully we can learn it today.
Are they?
Consider the old joke, "what do you call the medical student that graduated bottom of his class? 'Doctor'".
Same thing applies to sysadmins and IT. Sure there's plenty of skilled people working in facilities out of bumfuck, but also consider how few people want to work in the low-paid, low-respect IT jobs.
Those roles are often filled by the apathetic, mid-skilled, and close in proximity. They're connecting shit to the internet because their boss wants more data points.
What makes this particularly embarrassing is that power companies literally run thousands of miles of cables and could easily build a private network instead of using the Internet. The regulations should be amended to require it.
Again, STUXNET shows that bugs get into authorized devices really easily. One of the biggest issues is you have power guys that don't know dick about computer security using fleets of laptops that will need a huge amount of training to bring them up to the needed high levels of security.
More so there are thousands of these devices out in the field in power company trucks. All it takes is someone picking one of these up and infecting it for the problems to spread.
It is an insanely hard problem to solve. Breaches happen in military networks pretty often. As a measure of secrecy, they never tell you about it.
While you certainly make a good point about the difficulties that come with securing the laptops and other equipment, the military has the same problem. Most of the users of classified military systems are not themselves experts in computer security. The necessary training is not a graduate level course in security; it is training people to always match cable colors to port colors, to only use laptops provisioned by the organization, etc. The military disables, sometimes by physical means, USB ports on laptops that are used on classified networks to close one of the key security holes Stuxnet took advantage of.
Sure, it is challenging, but this is not impossible to solve. Utilities would have to be forced to do it and they would need time to overhaul their operations, but they certainly can do it, and there is an argument that the NSA's IA division should help (as the electrical grid is a critical for national security). It can be done, we just have to be willing to do it (and pay the cost of doing it).
They called it a "serious threat". This means they're not even counting it as a fully fulfilled harmful event, let alone one that could have irreversible consequences. At least you perceived it as potentially "painful".
So the zombie analogy stands. It's like people in New York calling the total annihilation of Raccoon City a "serious threat". The people of Raccoon City would beg to differ in their assessment. And likely others if the zombies escape containment.
I do agree with you both that it's better to plan for zombies before they appear, of course.
Now supermarket fridges are openly accessible over the Internet
I’m not sure it’s AI that’s the issue, it’s just a scapegoat for decades of bad security choices that nation state bad actors could have always exploited
Now that’s a perceived threat, ‘AI will do bad things’ is a great excuse for awful national infrastructure and security management
I know this sounds unusual, but I do think this wouldn't be the worst thing that could happen. Sure, it would cause a bit of chaos, but it might also make us think twice about continuing down our current path of unfettered development, and possibly even make us much more cautious, which might in turn even help us with the climate crisis.
Because human beings in a large mass only respond to one thing: immediate difficulty. They don't respond to delayed difficulty like "the climate is getting worse" or "technology will make life more difficult in the distant future". They only respond to immediate pain.
So I think it would be a good thing if the financial system had a serious threat. Then people with money, who tend to offset future problems with technical solutions and financial padding, would start to become much more suspicious of technology and defund AI and wasteful things that are likely to threaten their safety.