Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac...
by GuB-42 on Tuesday January 23, 2024 @08:19PM
(#39112413)
It is what you get when you ask ChatGPT for the US constitution, so yeah, AI generated.
This effect is described in the article. And depending on the context, it can be a feature rather than a bug. If you are using an LLM detector to check if a news article or student essay is "legit", then not only you don't want something from a LLM, but you don't want copy-paste plagiarism either. So for the purpose of checking legitimacy of supposedly original work, then it is a desirable kind of false positive.
I suppose simpler techniques can then be use to check for verbatim copies of famous text.
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac...
by comex on Tuesday January 23, 2024 @05:26PM
(#39110827)
The paper itself goes into detail about why the US Constitution and other memorized texts are misclassified. It’s surprising but not a killer flaw, since in most contexts it would only apply to direct quotes of famous texts.
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac...
by yreg on Tuesday January 23, 2024 @04:48PM
(#39110394)
Well that is not a novel text, is it?
Re:Spotting LLMs with Binoculars: Zero-Shot Detection of Mac...
by minimaxir on Tuesday January 23, 2024 @11:07PM
(#39113503)
I just added another tweet with the caveat others mentioned. (It was not an intentional omission)
It’s still not a nuance that most people trying to identify AI will respect even if they know it. Given that constraint, I really doubt the accuracy metrics as well.
Not very promising, though.