The specific concern that we have in DISARM:SIMC4 is that as AI systems start to be perceived as being smarter (due to being better and better at natural language rhetoric and at generating infographics), people in command will become more and more likely to set aside their doubts and just trust the computer, even if the computer is convincingly hallucinating.
This tendency of decision makers (including soldiers) to have higher trust in smarter-seeming systems is called Automation Bias.
> The dangers of automation bias and pre-delegating authority were evident during the early stages of the 2003 Iraq invasion. Two out of 11 successful interceptions involving automated US Patriot missile systems were fratricides (friendly-fire incidents).
https://thebulletin.org/2023/02/keeping-humans-in-the-loop-i...
Perhaps Stanislav Petrov would not have ignored the erroneous Soviet missile warning computer he operated, if it generated paragraphs of convincing text and several infographics as “evidence” of the reality of the supposed inbound strike. He himself stated that he felt the chances of the strike being real were 50-50. Even a slight increase in the persuasiveness of the computer’s rhetoric and graphics could have tipped this to 51-49 and thus caused our extinction.
Yet everytime there was a "real" attack, somehow the doctrine was not followed (in US or USSR).
It seems to me that the doctrine is not actually followed because leaders understand the consequences and wait for very solid confirmation?
Soviets also had the perimeter system, which was also supposed to relieve pressure for an immediate response.