Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks
by simonw on Thursday May 28, 2026 @10:15AM
(#48309256)
This study treats models disagreeing - returning both true and mostly true - as a failure.
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks
by pjdesno on Thursday May 28, 2026 @10:47AM
(#48309725)
They overstate their results in the headline.
In section 2, 34% of cases are found to have "substantive" disagreements differing by 2 or more buckets - True + Misleading, Mostly True + False, or True + False.
This is probably a better measure than the headline one. It's still a concerning fraction, although some fraction is no doubt due to forcing "I don't know" cases to return an answer anyway.
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks
by kostaj on Thursday May 28, 2026 @10:50AM
(#48309781)
Agree with @pjdesno, that the 34% substantive or polar disagreement might be a better headline number. Or even the 21% polar disagreement (at least one model True, and at least one model False), which is still high for many real-world applications.
The models were split between "true" and "mostly true". Given the "among the most" language either of those answers means effectively the same thing.
So the models were right? The actual criterion should be whether "Incomplete Egypt visa application forms" are indeed "among the most common reasons" or not.
That "true" and "mostly true" means effectively the same thing is irrelevant. It could just as well trip me up, and I'm a human. If somebody told me either answer, I'd still consider them right if the basic fact was right.