News for hackers. Stuff that matters.
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac... by vicgalle_ on Tuesday January 23, 2024 @04:44PM (#39110334)
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac... by GuB-42 on Tuesday January 23, 2024 @08:19PM (#39112413)
It is what you get when you ask ChatGPT for the US constitution, so yeah, AI generated.

This effect is described in the article. And depending on the context, it can be a feature rather than a bug. If you are using an LLM detector to check if a news article or student essay is "legit", then not only you don't want something from a LLM, but you don't want copy-paste plagiarism either. So for the purpose of checking legitimacy of supposedly original work, then it is a desirable kind of false positive.

I suppose simpler techniques can then be use to check for verbatim copies of famous text.

Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac... by comex on Tuesday January 23, 2024 @05:26PM (#39110827)
The paper itself goes into detail about why the US Constitution and other memorized texts are misclassified. It’s surprising but not a killer flaw, since in most contexts it would only apply to direct quotes of famous texts.
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac... by yreg on Tuesday January 23, 2024 @04:48PM (#39110394)
Well that is not a novel text, is it?
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac... by minimaxir on Tuesday January 23, 2024 @11:07PM (#39113503)
I just added another tweet with the caveat others mentioned. (It was not an intentional omission)

It’s still not a nuance that most people trying to identify AI will respect even if they know it. Given that constraint, I really doubt the accuracy metrics as well.