News for hackers. Stuff that matters.

A comment on Disagreement Among Frontier LLMs on Real-World Fact-Checks. Parent comment

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by jiggawatts on Thursday May 28, 2026 @08:54AM (#48308262)
Many of the rows in that spreadsheet reference "current events", which models aren't expected to do much better at than a human making an educated guess! They all have cutoff dates either last year or early this year and know nothing about what happened in "April 2026".

This is doubly problematic because you evaluated earlier models like Gemini Pro 3 instead of 3.1, GPT 5.4 instead of 5.5, etc...

Given that it's only a thousand short questions, you should be able to re-run your test in about an hour with the latest models, so... why haven't you?

Similarly, LLM output is non-deterministic, so if you could get more interesting stats of your data set by repeating each question 'n' times for each model.

[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by kostaj on Thursday May 28, 2026 @09:02AM (#48308378)
Two of the models used have retrieval capabilities and have access to newer information through search. The other three are parametric.
[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by simonw on Thursday May 28, 2026 @09:05AM (#48308440)
Comparing models with search tools to models without - when there's no option for "I am unable to answer this question without access to search" - doesn't make sense to me.
[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by kostaj on Thursday May 28, 2026 @10:19AM (#48309313)
Agree about comparing models with and without search capabilities. Even the two models with search capabilities (Sonar Pro and Gemini) agree only on 58% of the claims.
[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by furyofantares on Thursday May 28, 2026 @09:11AM (#48308511)
Yes, so in that case you set them up to disagree and then measured disagreement.
[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by throw310822 on Thursday May 28, 2026 @09:06AM (#48308453)
The title mention "fact-checks", but "fact checking" is a process in which facts are checked against sources, not one where you are given a random fact and have to tell if it's true or false from your own memory. That's what is normally called a quiz game. So a more honest title for this research would be "Models answer differently to quiz questions".
[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by jiggawatts on Thursday May 28, 2026 @10:18PM (#48318209)
> "Models answer differently to quiz questions".

from their future

It's as if you asked a human in 2026 to "fact check" something from 2027.

You're going to get an educated guess at best, a coin flip at worst.

[ Parent | on HN ]