"an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere"
and explains that it would also need to be goal-directed in a way which would be at odds with human wellbeing.
Stipulating all that, what is still missing is an explanation of the mechanism by which an AI, rogue or otherwise, could do harm. How is it supposed to affect the world outside its computing substrate?
Absent humans making available the interfaces and resources to do so, it can't. The referenced article includes an example of a genocidal human doing exactly that, and using an AI as a force multiplier. That, as the trope goes, is a social problem, not a technical problem, and it needs a social solution, not a technical one.
Each of the other examples in the referenced article (military AI going rogue, wireheading, amoral corporate AIs manipulating humans) require AIs interfacing with the physical world outside their computing substrate or with the biosphere. Again, because these scenarios remain dependent on humans making available such interfaces, I fail to see how a hypothesized "rogue" AI could achieve any autonomy to do serious damage.
I see this panic about rogue AIs as well-intentioned but misguided, and perhaps exploited by folks who would like to control / diminish / force licensing of general purpose computing.
True, but what more do you need than the ability to send web requests to arbitrary domains, and receive the responses?
Too many systems rely on people being unaware of exploits, but an AI would never forget something, or get bored or tired of trying, It doesn't need to be smarter than humans, just have enough persistence and attention to detail
This would be more reassuring if hooking it up to a Python prompt wasn't virtually the first thing people did.
How do you know the AI doesn’t have an interface to the middle silo?
Are there no military systems that are connected to the internet?
Is the power grid connected to the internet? How about our water filtration system?
I’m not worried that gpt4 is going to start launching missiles, but I also don’t understand where the certainty comes from that it isn’t possible for an internet connected AI to launch missiles.
Cool SciFi showerthought:
An AI will probably know that Stuxnet happened.
https://en.m.wikipedia.org/wiki/Stuxnet
An AI could trick multiple human actors into unknowingly assembling all the pieces together for a nuclear launch.
Can someone please explain to me what exactly the danger is / the dangers are of "super-intelligent" AI?
AIUI, an AI is a combination of hardware, software and parametrization. In broad terms, it exists as a black box which supplies to humans responses to token sets fed to it.
Even if an AI has the launch codes for ICBMs somewhere in its training data, it doesn't have an interface to the nearest missile silo to use them. It cannot commandeer the resources (hardware, space, cooling, electricity) it needs to operate, it is dependent on humans to supply those. So humans can pull the plug on it at any time.
Even if an AI were to become both sentient and nefarious, by what mechanism would it harm humans?
I'm genuinely looking for concrete examples of such a mechanism because I can't imagine any which humans couldn't trivially control or override.