Hm, maybe a Boston dynamics robot could drop a USB in front of NORAD?

No need, just hijack their Uber Eats order. Apparently delivery people can walk right in and find soldiers napping on the job.

The risk is relying on AI to give a human information on whether or not to push the button and the AI gets it wrong but communicates so effectively the human trusts it without question.

An AI learning about and then executing a social engineering attack on a human (whether with malicious intent or not) seems like a reasonable thign to be worried about to me.

Initiating a nuclear strike requires sending authentication codes via designated secure channels. No matter how good some hypothetical future AI gets at impersonating President Biden, it won't be able to ring up the duty officer at Minot Air Force Base and order them to nuke Moscow. Don't believe what you see in the movies.

I don't know, I'm pretty sure some future AI is up for the challenge. I can almost picture an AI telling another AI, hold my beer...

imagine instead just making it look like someone is attacking you

such incidents stemming from human error have already gotten us close to responding with force