How's your hypothetical any different now than it would've been in the past 15 or so years of Google Translate's existence?
Translation software predates Google translate, and I'm not claiming something has suddenly changed. It's slowly gotten better, and I assume the temptation to only pretend to have human translation will keep growing with it. The safer a scam seems, the more people will try it.
There is also the translaxy bot on poe.com which i use to translate english or modern greek to ancient greek. Out of this world good translation.
I mean, are humans still employed to translate text? Like an employee doing that job, and only that?
Another example would be highly specialised, industry-specific texts with a lot of jargon, for which professional translation offices get you translators who aren't just fluent in the languages, but also knowledgeable about the area the text is about.
Understanding the output of an LLM is similar to the output of a translater.
If the recepient doesn't/can't understand it, all bets are off.
Say you don't understand python but have an LLM write some for you, but you have no way of knowing what it's doing.
What if you have a malicious LLM hosted somewhere and it writes malware insatead of what you asked for.
If you don't understand the output you end up with, you run it and it pwns your network.
https://news.bbc.co.uk/2/hi/7702913.stm