Comparing models with search tools to models without - when there's no option for "I am unable to answer this question without access to search" - doesn't make sense to me.

Agree about comparing models with and without search capabilities. Even the two models with search capabilities (Sonar Pro and Gemini) agree only on 58% of the claims.

Yes, so in that case you set them up to disagree and then measured disagreement.

The title mention "fact-checks", but "fact checking" is a process in which facts are checked against sources, not one where you are given a random fact and have to tell if it's true or false from your own memory. That's what is normally called a quiz game. So a more honest title for this research would be "Models answer differently to quiz questions".

> "Models answer differently to quiz questions".

from their future

It's as if you asked a human in 2026 to "fact check" something from 2027.

You're going to get an educated guess at best, a coin flip at worst.