News for hackers. Stuff that matters.

A comment on Disagreement Among Frontier LLMs on Real-World Fact-Checks. Parent comment

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by Someone on Thursday May 28, 2026 @09:22AM (#48308610)
For those questions, it wouldn’t surprise me at all if five well-educated intelligent humans disagreed on over two out of three of them.

I would answer “don’t know” on many, but that’s not an option.

[ Parent | on HN ]
Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by kostaj on Thursday May 28, 2026 @09:32AM (#48308747)
Yes, inter-human-annotator disagreement is also high on similar type of questions (AVeriTeC) - inter-panel agreement: κ=0.619. Tried giving the models a fifth option, Abstain, but some models seem to use it to "avoid answering hard questions" more than others.
[ Parent | on HN ]