"It wasn't me, it was my AI" is definitely going to be a nightmare for a while.
In this example, the equivalent would be everyone gets their AI models banned vs. putting the perpetrator of this fraud in jail.
We have regulations and requirements and licences and insurance in most of the civilized world for dangerous stuff that is intended for public use. Maybe AI should also have some of that.
I know you were referencing AI, but this problem predates AI by decades.
Go open up any news website, newspaper, or turn on the TV.
"Man struck by speeding car and killed"
"Vehicle plows into pizza shop"
"Truck takes out telephone pole"
...and these days even bystanders will blur plates in the photos they post. The police won't release any info about the driver, the news won't either.
Listen to friends, family, coworkers talk.
"I can't believe that car just ran that red light!" "The other day I was almost hit by a car in the crosswalk." "All those cars honking their horns late at night are keeping me awake."
Etc.
Once you see it, you can't unsee just how thoroughly the auto industry has managed to transfer perception of responsibility from the operator to the object.
See the last years of the industry constructing bigger and bigger tank-ish SUVs and pickups, the more aggressive the better.
One of these days, they will start putting stuff copied from Carmageddom, and then fake surprise as someone causes a carnage in a road rage incident.
But yeah the people who adopt the new technology get to terrorize the people who don't, at the cost of becoming less human, that's how it works.
They also shared the poorly anonymized messages it received from target "customers" who complained about the unsolicited invoices. On example of this poor anonymization is removing the sender's username but leaving the domain name, when the domain name is, for example, a personal domain for a single person.
This is like someone who removes a brake pedal from a tractor, uses a stick to hold down the throttle, and lets it loose on his field. When it leaves the field and runs someone over, that is criminal negligence.
If the prosecution is able to prove beyond a reasonable doubt that the person gave a prompt that was intended to commit a crime, then of course we can prosecute them for that. The AI is just a tool to commit fraud at that point, and is no different than a person who uses photoshop to alter a check to commit fraud.
Of course, it'd be better to not regulate, keep LLMs users reponsible and publicize this reponsibility in order to mitigate damage. But if this is not enough then we will have to move the needle somehow. Similarly to guns, drugs and so on.
That doesn’t mean you can get away with anything as long as you don’t intend to commit a crime; there are many other crimes that don’t require intention at all, like involuntary manslaughter or gross negligence.
My point is that if you want to charge someone with a crime that requires intention, you have to prove that intention.