Why is this true?

I think what will happen is that there effectively be an arms race between the bot generation and detection sides. As anomalies are discovered in the distribution of content generated by bots vs humans, bots will update to remove the anomaly. And researchers will try to find new anomalies. In the end as bots get better, the difference in the two distributions will be indistinguishable in practice.

Possible, sure. Likely, maybe. But you made it sound inevitable? Of that I am less sure.

For any accessible detector you can automate the evasion. E.g, crudest variant would be:

   generate text
   run it through the detector
   finetune model w/result of the detector
   repeat until detector fails to detect.

But every time the failure rate of the detector starts getting high, the other side has a lot of development work to do finding new patterns that won't also lead to lots of false positives.

So while getting a point where it's totally impossible to detect may well take a long time, it'll be periods of the detector suddenly being hard to bypass for a while, and gradually getting easier, repeat. It's highly unlikely it'll ever be impossible to bypass for any length of time.

The only way of keeping such a detector working for any length of time will be to make it expensive or very hard for adversaries to get access to, which means effectively relying on only giving access to trusted users.

Isn't this basically how autoencoder models are trained?

This assumes the ability to perfectly model this function, which is in a sense what we're setting out to demonstrate. It can certainly not be taken for granted.

Firstly, it doesn't need to perfectly model it. It needs to model it to within a range where claiming to have detected it causes more false positives than acceptable, and given the range of quality of human output that's a pretty wide span.

But secondly, LLMs wired up with a feedback loop and and appropriate IO can model Markov models (models, not chains) which are Turing complete, so a LLM wired up this way is clearly Turing complete as well. As such it's a question of training, not whether or not the model can model it.

Because eventually LLM based AI will be advanced enough to author its text with as much variety and diversity as any human, making it indistinguishable from anything authored by a human.

That feels like a _huge_ assumption. It would arguably be quite surprising just based on the prior history of ML-y stuff; there's a tendency for things to get 95% good enough quickly, and then never quite make it to 100% (because 100% is really a completely different sort of problem, arguably requiring AGI).

My favourite example of this: speech recognition. In the 90s this went quite quickly from a joke to a tech demo that required enormously expensive hardware to a tech demo that you could run on your laptop to actually marginally useful (you may remember Apple and particularly MS claiming that it was the future, and would soon be the primary way that people interacted with computers).

And then, well... like, voice recognition today is certainly better than Dragon Dictate was 25 years ago, but you probably still wouldn't want to trust it with anything important, and it's certainly far worse at it than a human is.

I'm struggling to think of _any_ ML-ish thing in the human-imitating space which has gotten all the way to virtually perfect/as good as human. Maybe OCR? Though even then, you still sometimes get odd results.

>I'm struggling to think of _any_ ML-ish thing in the human-imitating space which has gotten all the way to virtually perfect/as good as human.

Go?

Oh, yeah, actually, fair, a few games. Though, can an expert human player reliably determine that they're playing against a machine? I suspect they can, but I don't know.

In general the goal of these progrms is to win, not to pretend to be human.

In chess, there is a project called Maia which aims at predicting the human move rather than the best move. Even then it blunders less than humans of a similar rating, so it can still be detected.

https://maiachess.com/

> That feels like a _huge_ assumption.

Huh, seems like a very reasonable inference to me.

Will it?

I think so or I wouldn't have said so. You obviously disagree which is why you're obtusely indicating that.

Guess we'll see.

Because the LLM can train itself against the AI detector to become undetectable.