Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks
by simonw on Thursday May 28, 2026 @10:15AM
(#48309256)
This study treats models disagreeing - returning both true and mostly true - as a failure.
[ Parent
| on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks
by pjdesno on Thursday May 28, 2026 @10:47AM
(#48309725)
They overstate their results in the headline.
In section 2, 34% of cases are found to have "substantive" disagreements differing by 2 or more buckets - True + Misleading, Mostly True + False, or True + False.
This is probably a better measure than the headline one. It's still a concerning fraction, although some fraction is no doubt due to forcing "I don't know" cases to return an answer anyway.
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks
by kostaj on Thursday May 28, 2026 @10:50AM
(#48309781)
Agree with @pjdesno, that the 34% substantive or polar disagreement might be a better headline number. Or even the 21% polar disagreement (at least one model True, and at least one model False), which is still high for many real-world applications.