How many instances of risk are there in the above literally factual story? What is the nature of this risk? Is there any hidden risk, might there be more than meets the eye?
I'd like to distinguish between "fully autonomous luxury doomsday" scenarios, where it's just the AI and some robots and no humans in the loop, vs "accelerationist" scenarios where humans are in the loop and are choosing to be complicit in whatever happens because they believe they will benefit from it. Because we can't solve the human alignment problem.
Who/that which has most control over the beliefs of the most humans (with some weighting) is the most powerful character. You can build the most powerful AI you want, but whoever / whatever is able and willing to invoke Total War (or invoke mass human action toward a particular goal) can Win the Game (or, destroy the playing field so it no longer matters).
For example:
> Because we can't solve the human alignment problem.
Whoever or whatever put this sort of thinking into the minds of 99% of humans, or knows how to exploit it (see: Trump) is very, very powerful. Someone or something that could displace it would be even more powerful, maybe even the most powerful.
No offense intended btw, I think it is fairly inevitable, especially when moving fast.
Everything dies if you drop enough nukes on it.
(This is also our insurance plan against humans trying to destroy the world)