Simplified operations, faster reaction time, eliminating human resistance for obeying killing orders. See "War Games" [0] for a hypothetical exploration of the concept.
> a chatbot.
Some claim it's self-aware. Some say it called for airstrikes. Some say it gave a hit list for them. It might be a glorified Markov-chain, and I don't use it, but there's a hoard of people who follows it like it's the second Jesus, and believe what it emits.
> I can't see it happening.
Because, it already happened.
Turkey is claimed to use completely autonomous drones in a war [1].
South Korea has autonomous sentry guns which defend DMZ [2].
[0]: https://www.imdb.com/title/tt0086567/
[1]: https://www.popularmechanics.com/military/weapons/a36559508/...
Speaking of which, it's perfectly legal for a civilian to own a fighter plane such as an F-16 in US and many other countries. You just have to demilitarize it, meaning no weapon pods.
The reason this isn't an issue in practice is because such repurposing would require significant intelligence/electrical engineering skill/etc. The point is that intelligence (the "I" in "AI") will make such tasks far easier.
>Ten-year-old about to play chess for the first time, skeptical that he'll lose to Magnus Carlsen: "Can you explain how he'll defeat me, when we've both got the same pieces, and I move first? Will he use some trick for getting all his pawns to the back row to become Queens?"
Assuming that you're not running this query over an API and relaying these answers to another control system or a gullible operator.
An aircraft control computer or reactor controller won't run my commands regardless of its actuators connected or not. Same for weapon systems.
Hall pass given to AI systems just because they're outputting text to a screen is staggering. Nothing prevents me to process this output automatically and actuate things.