How we measured AI writing across arXiv, and where the measurement breaks

Re:How we measured AI writing across arXiv, and where the me... by paxys on Monday July 20, 2026 @02:20PM (#48982750)

What detector are you using? How can you be sure of its accuracy given that every commercial AI detector has been debunked?

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by JamesBarney on Monday July 20, 2026 @02:29PM (#48982888)

Almost every. Pangram is pretty accurate on longer texts. I haven't seen any glaring examples of false positives.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by adamgordonbell on Monday July 20, 2026 @09:04PM (#48986902)

This! Pangram is very good. They claim 1 in 10,000 fp rate.

I had to change my mind on AI detectors after playing around with it.

It would be interesting to hear how this detector compares. It also seems to be aiming for low fp rate.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by malshe on Monday July 20, 2026 @11:08PM (#48987589)

I’ve heard good things about Pangram but it’s trivially easy to fool it. I generated text using Opus 4.8 and then humanized it using Grammarly. Pangram determined it 100% human-written. Tried it numerous times with the same result. GPTZero claimed with 80% confidence that it was human-written but AI polished. Nothing in that text was human-written.

[ Parent | on HN ]

Re:How we measured AI writing across arXiv, and where the me... by JamesBarney on Tuesday July 21, 2026 @12:09PM (#48994228)

How long was the text? And mind sharing?

[ Parent | on HN ]