In particular, conference papers are already extremely formulaic, organized in a particular way and using a lot of the same stock phrasings and terms of art. AI or not, it's hard to tell them apart.

Its the reviews that were found to be AI, not the papers themselves. The papers were just 1% AI according to the tool, so it seems to work properly.

> AI or not, it's hard to tell them apart.

Apparently not for this tool.

Co-founder of Pangram here. Our false positive rate is typically around 1 in 10,000. https://www.pangram.com/blog/all-about-false-positives-in-ai....

We also wanted to quantify our EditLens model's FPR on the same domain, so we ran all of ICLR's 2022 reviews. Of 10,202 reviews, Pangram marked 10,190 as fully human, 10 as lightly AI-edited, 1 as moderately AI-edited, 1 as heavily AI-edited, and none as fully AI-generated.

That's ~1 in 1k FPR for light AI edits, 1 in 10k FPR for heavy AI edits.

Give your final sentence a re-read there....

Thanks, fixed.

The conference papers were 1%, peer reviews 20%, is there another reason for that big difference than more of the peer reviews being AI generated than the papers themselves?

We can't use this to convict a single reviewer, but we can almost surely say that many reviewers just gave the review work to an AI.

Peer reviews are much shorter than papers. A bad review may only be six to eight sentences along the lines of "Math is confusing" and "Too little related work". 20% AI generated seems high if for no other reason than writing a bad review takes very little effort.