But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.
The rule of law prevented WWI and WWII, right? Oh, no it did not, tens to hundreds of millions died due to human stupidity and violence depending on what exactly you count in that age.
> Both criminal, terrorist, groups at war
Human organizations, especially criminal organizations have deep trust issues between agents in the organization. You never know if anyone else in the system is a defector. This reduces the openness and quantity of communication between agents. In addition you have agents that want to personally gain rather than benefit the organization itself. This is why Apple is a trillion dollar company following the law... mostly. Smart people can work together and 'mostly' trust the other person isn't going to screw them over.
Now imagine a superintelligent AI with a mental processing bandwidth of hundreds of the best employees at a company. Assuming it knows and trusts itself, then the idea of illegal activities being an internal risk disappears. You have something that operates more on the level of a hivemind toward a goal (what the limitations of hivemind versus selfish agents are is another very long discussion). What we ask here is if all the worlds best hackers got together, worked together unselfishly, and instigated an attack against every critical point they could find on the internet/real world systems at once, how much damage could they cause?
Oh, lets say you find the server systems the super intelligence is on, but the controller shuts it off and all the data has some kind of homomorphic encryption so that's useless to you. It's dead right? Na, they just load up the backup copy they have a few months later and it's party time all over again. Humans tend to remain dead after dying, AI? Well that is yet to bee seen.
Drawing an artificial line between you and the danger is a great way to find yourself in a Maginot Line with AI driving right around it.
"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.
The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").
So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.
It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).
But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.
Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.
The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.
There's a good counterargument here[1] that seems reasonable: "One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"
Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.
>AI is an application of math.
It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.
And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.
Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.