https://upload.wikimedia.org/wikipedia/commons/d/de/Photosho...
You should try photocopying money some time.
https://www.grunge.com/179347/heres-what-happens-when-you-ph...
The core of the issue is that there are many people, including regulators, who wish that software did exactly that.
"Charles II had re-turned to the English throne in 1660 and was appalled at the state of printing in his realm. Seditious, irreligious, pernicious, and scandalous books and pamphlets flooded the streets of London (among them the works of Milton and Hobbes)...[He] required that all intended publications be registered with the government-approved Stationers’ Company, thus giving the king his “royal prerogative”—and by extension, giving the Stationers the ultimate say in what got printed and what did not.
...it is not surprising to learn that the 1662 Act only met with partial success. One gets the sense that London in the late seventeenth century was a place where definitions of morality were highly subjective and authority was exercised in extremely uneven fashion."
https://dash.harvard.edu/bitstream/handle/1/17219056/677787....
“Blood and soil” and all that.
Humans are notoriously bad at detecting intent, because we're wired to be supportive and helpful...which is why social engineering is becoming one of the best methods for attack. And this kind of attack (in all its forms, professional or not), is one reason why some societies are enshittifying: people have no choice but to be persistently adversarial and suspicious of others.
As for AI, I think it's going to be no better than what you end up with when someone tries to "solve" this problem: you end up living in this world of distrust where they pester you to check your reciept, have cameras in your face everywhere, etc.
How do you defend against that? I'm not sure you do... A tool is a tool. I wouldn't want my CAD software saying, "I think you're trying to CAD a pipe bomb so I'm going to shut down now." Which I think turns this into a liability question: how do you offer up a model and wash your hands of what people might do with it?
Or... you just don't offer up a model.
Or... you give it the ol' College try and end up with an annoying model that frustrates the hell out of people who aren't trying to do any evil.