Re:ARC-AGI Leaderboard
by pierrefermat1 on Saturday July 25, 2026 @09:12AM
(#49047332)
What seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution .
Simple/common tests is not an explanation for why only now Opus 5 is the only model encoding the answers like this. Something like the holdout test suite being leaked or Anthropic cheating (e.g. 'accidentally' including previous hold out run data in Opus 5 training) makes a much stronger fit.
It may read information about the benchmark, such as on blog post, or Twitter feeds (example OP) without active cheat
They state the puzzle is "Witness-like" which I assume means that it follows the rules from the well-known puzzle game "The Witness" which Opus definitely knows.
Re:ARC-AGI Leaderboard
by rbuccigrossi on Saturday July 25, 2026 @08:12PM
(#49053210)
No, I believe you are misunderstanding the quote. Each “question” in ARC-AGI-3 is a game that has hidden rules that you can understand if you look at the game board long enough. This quote means that Opus 5 is looking at the game board, figuring out the rules, and writing out the rules before it makes a single move. You can do the same thing if you go to the ARC-AGI-3 website and try some of the games.
I guess there is no way this can happen without benchmark being part of the training data??