As usual, journalists trying to explain what scientists do and misrepresenting the facts.

The paper mentions accuracy i.e. (true positives + true negatives) / total examples. And it's actually 100% accurate i.e. there are no false positives _or_ false negatives.

But the big caveats are:

1. this was tested only on 180 examples, which is a very very small dataset to draw conclusions on, and

2. this is obviously an adversarial space so any classifier will be obsolete with the next training run

I'm bearish on any attempt to distinguish real content vs. AI generated content (on any medium, text, image or anything else). This is an adversarial game and the AIs can incorporate your fancy algorithm to fool you better. In the end these projects only end up improving the AI models in terms of realism.

> 1. this was tested only on 180 examples, which is a very very small dataset to draw conclusions on, and

If you have 180 samples, and a >99% accuracy (meaning a single misprediction), that is a statistically significant conclusion with a p-value of 99.994%.

Furthermore, from the article it was only whole scientific papers written ChatGPT in which they attained 99%; at the "paragraph level" they claim 92%, which means the claim of "AI generated content" in the headline is overly broad IMHO

You got me. As a large language model I just don't stand a chance against a similarly trained and reliable large detection model.

I used your algorithm on your post. Turns out, your comments are written by AI!

It's fool-proof!

This reply falls under the RT;PAM category (read title; posted angry comments).

/s