~Alan Watts…
The space of all possible mathematical worlds, free to explore and to play in.
It is infinitely more expressive than the boring base reality and much more varied: base reality is after all just a special case.
From time to time the Minds have to go back to it to fix some local mess, but their hearts are in Infinite Fun Space.
~Iain Banks
But larger than any of this is that if we're dealing with super intelligent AI, we'll have no common frame of reference. It will be the first truly alien intelligence we will interact with. There will be no way to guess its intentions, desires, or decisions. It's smarter, faster, and just do different to us that we might as well be trying to communicate with a sparrow about the sparrow's take on the teachings of Marcus Aurelius.
And that's what scares me the most. We literally cannot plan for it. We have to hope for the best.
And to be honest, if the open Internet plays a part in any of the training of a super intelligent AI, we're fucked.
Yeeeeess, but the inverse is also true.
Thing is, we've had sufficiently advanced robotics for ages already — decades, I think — the limiting factor is the robots are brainless without some intelligence telling them what to do. Right now, the guiding intelligence for a lot of robots is a human, and there are literal guard-rails on many of those robots to keep them from causing injuries or damage by going outside their programmed parameters.
The reason AI won't destroy us for now is simple.
Thumbs.
Robotic technology is required to do things physically, like improve computing power.
Without advanced robotics, AI is just impotent.