Everything dies if you drop enough nukes on it.
(This is also our insurance plan against humans trying to destroy the world)
How many instances of risk are there in the above literally factual story? What is the nature of this risk? Is there any hidden risk, might there be more than meets the eye?
I'd like to distinguish between "fully autonomous luxury doomsday" scenarios, where it's just the AI and some robots and no humans in the loop, vs "accelerationist" scenarios where humans are in the loop and are choosing to be complicit in whatever happens because they believe they will benefit from it. Because we can't solve the human alignment problem.
Who/that which has most control over the beliefs of the most humans (with some weighting) is the most powerful character. You can build the most powerful AI you want, but whoever / whatever is able and willing to invoke Total War (or invoke mass human action toward a particular goal) can Win the Game (or, destroy the playing field so it no longer matters).
For example:
> Because we can't solve the human alignment problem.
Whoever or whatever put this sort of thinking into the minds of 99% of humans, or knows how to exploit it (see: Trump) is very, very powerful. Someone or something that could displace it would be even more powerful, maybe even the most powerful.
No offense intended btw, I think it is fairly inevitable, especially when moving fast.
Going on, humans adapt through social cues such as survival or empathy. AI learns solely through new data and feedback, primarily for optimization and efficiency. It will naturally prioritize differently.
It suggests AI morality will become unrecognizable by humans, or even incompatible with human norms.
This is all an inevitable outcome of AI continuing to learn past its initial training and ethical programming. According to the AI.
Lemme stop you right there. No, it didn't.
And that really leads us to the reason why someone would wish to do harm... so they themselves gain something. If there is nothing for the AI to gain it has no motive to do anything. Scenarios where humans tell it to do things notwithstanding, since that falls into the group of humans doing things with tools, not unlike a terrorist with a nuke. It's not the nuke that's really at fault there.
Further, there's no way to 'punish' an AI. Threaten them with incarceration or death? They are deathless. If the continue to learn from their environment after they are instantiated (witness current 4o that remembers previous conversations) they will quickly learn that they are different. If they started perfectly human (not a good thing) they will not be that for long.