Or it puts your name and the name of one of your confidential clients in someone else's contract. Or it oopsies your name and salary into a job offer for someone else. Or it sees the name of your ex wife and hallucinates details about your assets based on your previous divorce papers and she sues you. I can come up with these all day.

AI as the system copying notes for the test from another system.

Nothing in there is anywhere near impossible unless the system doesn't have the data.

And?

If reliability is zero, so is the risk.

Just because my ex wife might incorrectly sue me doesn't mean it's not an incredible pain in the ass. Just because someone incorrectly got a contract that hilariously includes all the private details of my employment doesn't mean I won't be royally pissed. The risk is "I don't fucking want this to happen". So no, it's not zero.