News for hackers. Stuff that matters.

A comment on Disagreement Among Frontier LLMs on Real-World Fact-Checks. Parent comment

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by simonw on Thursday May 28, 2026 @09:33AM (#48308757)
What's the correct answer for "During a private Saturday call, Democratic members of the United States House of Representatives from Virginia and Hakeem Jeffries discussed strategies after losing a redistricting case at the Supreme Court of Virginia, including trying to flip two or three Republican-held seats under the existing map."?

You can only say True, False, Mostly True or Misleading.

(And you're not allowed to search for information.)

[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by kostaj on Thursday May 28, 2026 @09:53AM (#48308994)
Search was enabled for 2 of the 5 models -- Gemini and Sonar Pro. The disagreement between them is still high - different verdict on 42% of the claims. Fully agree, that some of those claims are hard to classify for a human as well -- the real-world messiness...
[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by dcreater on Thursday May 28, 2026 @10:25AM (#48309390)
Why was it enabled for only 2 of the 5?

Other burning questions: What methodology was used to choose the question set? Why not allow explanations? How many passes were done for each LLM?

[ Parent | on HN ]