Look at any of the "flash crashes" of recent years were out of control AI escalated the problem much faster than humans could react to it. Just take that and imagine that there are nuclear weapons involved.

Ok. None of those have had any long term negative outcomes. What was I supposed to imagine?

A stock dipping isn’t so bad because it can go back up. A nuclear missiles cannot unexplode.

And professionals price the downside into those trades. You don’t engineer trades the same way you do nukes cause it literally doesn’t matter if you get them wrong. As the history of “flash crashes” has shown.

A Bit easier to recover from a market dip than a nuclear strike don't you think?

It wasn't even that you recovered from the dip, the trades were actually busted (invalidated.)

Can't unexplode a bomb, for sure.

Trades are not routinely busted during flash crashes.

They were in "the" flash crash.

It had some pretty long term negative effects for Knight Capital.

Knight didn’t suffer from a flash crash. They suffered from a operational risk incident. This is literally something that professionals price into their trades. The long term negative ramifications were nothing. The next day a new firm was running the same trades under the same name.

I thought Knight got bought out a year later?

And for those few failures think of the trillions and more of the decisions that machines made that did not harm, but made lives better than before machines were doing work for us.

We understand how people work and think pretty well, their edgecases, and countermeasures to said edgecases.

We don't understand edge cases of AI much at all, and the terrain-space of "possible minds" they might be is vastly broader than that of a human mind, and utterly outside our experience.

An AI might work with absolute 100% flawless precision for 100 years, then decided one day to kill everyone. A lot harder to see that coming from a totally alien intelligence than a human one. AI does not think like humans, have human values, or care about what humans care about, that might make them extremely unpredictable and dangerous.

There will be no time for iterative development, or learning from our mistakes, a sufficiently advanced AI knows we would try to stop it, and make sure that we never can.

https://www.youtube.com/watch?v=tcdVC4e6EV4