News for hackers. Stuff that matters.

A comment on Disagreement Among Frontier LLMs on Real-World Fact-Checks.

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by throw310822 on Thursday May 28, 2026 @08:58AM (#48308323)
Not sure I'm understanding this. The models are asked to evaluate the truth of random claims out of their own head (except for Gemini with search grounding)? Isn't it exactly the same as asking people to play any quiz game and then rating them as "they disagree n% of the time"?

The output buckets are also pretty questionable- the difference between "True" and "Mostly true" is pretty fuzzy. Is this marked as a "disagreement"?

[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by kostaj on Thursday May 28, 2026 @10:57AM (#48309885)
Agree that True and Mostly True might be very close and could be a calibration difference. Misleading and False, as well. A better headline number might be the 34% claims with substantial or polar-opposite verdicts.
[ Parent | on HN ]