Doesn't have to be "safe" / "not dangerous" (a lot of weapons systems are very dangerous in different ways) but it should be predictable, using some reasonably simple mental model.
"Refusing to deploy" might not be a sensible option if your enemy is willing to deploy.
It seems safe to predict that human in the loop systems will be quickly overwhelmed by fully autonomous systems. The tradeoff may end up being having to balance the risk of getting killed by your own systems versus being killed by your enemy's systems, because yours were too "tame".
In the AI ethics arena, this is really one of the best arguments I've heard for the military to pursue and have automated systems in place. US leadership may agree and see all the issues with using killer robots but other countries are developing the same technology and may have less ethical concern and considerations then you do. They may be willing to let these systems fail and kill innocent bystanders or even their own soldiers. They may even look at these and consider it a necessary loss to move forward and succeed in their goals. As such, if you're not willing to make such compromises as well, unless your approach ultimately achieves the goal or you have other techniques to mitigate these corner cutting strategies, you too must cave.
If China decides to engage in conflict and releases killer robots and we don't have killer robots, can we contend? If we can't, we might have to engage in the race to the bottom that is competition unless we have an alternative mitigation strategy that doesn't require us to compromise our ethics.
I imagine this is the dilemma scientists faced during WWII when developing nuclear weapons. Part of the argument that lead to nuclear weapons was that Germany was already working on the process. This resulted in other countries engaging and ultimately building successful weapons where Germany failed. Ultimately, we now live in a world where lots of people have nuclear weapons... and overall it's been the proverbial Mexican standoff with nuclear deterrence strategies.
Is there an AI deterrence strategy? Nuclear worked because it lead to the case of a global collapse in terms of what nuclear war would bring. Killer robots that aren't sentient don't seem to have quite the same threat and seem like militaries could utilize them as they fall in an ethical grey area globally and have far less risk of ending humanity on a global scale unless we start talking about sentient terminator killer robot level uprising which is not even remotely where AI is currently or in the foreseeable future.
All of these systems are going to be as classified as possible for any country developing them, so we’d only find out once on or more of the following become true;
- it’s several decades after they’re out of date
- a politician needs a major scary saber to rattle for some reason
- they get used in a shooting war where the public can see them.
Neither of which favor us knowing current capabilities of any of the major players right now, except MAYBE Russia.
For many black box systems, the average performance may be comparable or better than humans, but when errors occur, they should fail predictably.
If systems fail unpredictably (ie unbounded in the types of errors it makes), then they can never be deployed in the field without humans in the loop.
This is why self driving cars can, at the same time, be on average safer but still not be accepted by public.
They're not safer, though.
or more realistically, your autonomous version of an aerovironment switchblade launches, has image recognition/onboard trained neural network that can identify a russian spec T-72 tank or BMP or similar, doesn't see any, loiters for a while and then self destructs.
Imagine launching your autonomous weapons, having them scan the area for enemy targets, only see you, conclude that you would not have launched if there were no enemies so obviously the one and only target it sees must be an enemy, and turns around and kills you.
After that happens just a few times, the other soldiers would revolt and refuse to use the weapon system anymore. That is the sort of thing that motivates having humans in the loop, even if on average there would be greater safety without humans in the loop.