What detector are you using? How can you be sure of its accuracy given that every commercial AI detector has been debunked?

Almost every. Pangram is pretty accurate on longer texts. I haven't seen any glaring examples of false positives.

This! Pangram is very good. They claim 1 in 10,000 fp rate.

I had to change my mind on AI detectors after playing around with it.

It would be interesting to hear how this detector compares. It also seems to be aiming for low fp rate.

I’ve heard good things about Pangram but it’s trivially easy to fool it. I generated text using Opus 4.8 and then humanized it using Grammarly. Pangram determined it 100% human-written. Tried it numerous times with the same result. GPTZero claimed with 80% confidence that it was human-written but AI polished. Nothing in that text was human-written.

How long was the text? And mind sharing?

Could you share some pre LLM false positives? Would be interesting to see what is tripping the detection. Did a tiny fraction of authors write like LLMs, before LLMs?

This is a pretty stunning result. The time series looks really convincing. Is the way the detector itself is trained orthogonal to this or could there be some "leakage" in that the pre-chatgpt text is in the (positive) training data?

I tried my best to avoid leakage. If you're curious about how I trained the detector I have a writeup on it: https://unslop.run/blog/how-our-ai-text-detector-works

FYI this is all relatively new so there might be lots of issues and iterations coming.

My honest first impressions, since I think the project is well intended: This writeup is itself AI, and I would venture to call it slop. The Calibration section is borderline uninterpretable, and I challenge any non-author who claims to understand it to answer some basic peer review questions about it.

I have to concur, unfortunately. This isn't reliable, reproducible, or interpretable.

Extend it to scan papers from pre-2020. That should give you a better baseline accuracy for your detection system.

this is neat! is the model available somewhere for local execution? or even a lookup table with your results for all arXiv pre-print codes? I want to run it on lots of pre-prints and I don't want to kill your server

Thank you, I plan to release the arxiv preprint codes. Also don't worry about killing my server, let me know if you succeed :D

Wouldn't that provide an excellent signal to tune models to be less like AI? I suppose that's a good thing ultimately.

Thank you for this work.

A signal, I put an AI generated of mine (with heavy human guidance on aesthetics mostly and some minor human editing) and got 5% only.

Couple of methodological notes

* pre-chatGPT is not an effective control because language evolves. In the arxiv corpus in particular there are "fashions" in research depending on what gets funded lately, not to mention many new words and topics not invented before a given paper.

* In general, detecting AI from content seems difficult as humans write like they read. To the extent there are unique factors recognizable as AI and to the extent humans read them, they will eventually incorporate them into their writing style. Accordingly, you'd need to model a rolling window of "AI tells" that decay at some rate.

Have you considered trying the same sort of analysis between two time periods pre-ai?

What I mean is, are there statistical differences in how people write that would give similar looking results between say 1990-2000 and 2000-2008? Are we just detecting the natural progression of language here?