Disagreement Among Frontier LLMs on Real-World Fact-Checks

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by segmondy on Thursday May 28, 2026 @09:18AM (#48308572)

Yup, if anything this should be a guide on how not to eval a model. Furthermore, let's say the labels were non ambiguous, why would we care about alignment between the models? The only number I would personally care about is percentage of correct answers so I know which models to pick. I reckon with clear and non ambiguous prompts that we would see huge agreement if not 100% on real world facts. The huge models are scary good in their world knowledge.

[ Parent | on HN ]

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by kostaj on Thursday May 28, 2026 @09:24AM (#48308638)

This paper covers only the disagreement between models and established only the floor of the error, based on the disagreement, but not which model is better. Planning to follow up with another study to benchmark against human-labelled verdicts still using a corpus that the models have not seen during training.

[ Parent | on HN ]

Re:Disagreement Among Frontier LLMs on Real-World Fact-Checks by aspenmartin on Thursday May 28, 2026 @10:38AM (#48309592)

You also need to involve better measures of agreement that are standard in the literature like krippendorfs alpha with ordinal metric. So many footguns in this methodology

[ Parent | on HN ]