News for hackers. Stuff that matters.
Re:ARC-AGI Leaderboard by andrepd on Saturday July 25, 2026 @09:19AM (#49047367)
I'm shocked, astonished even, that enterprises on which trillions of dollars are being poured would consider cheating on marketing benchmarks.
Re:ARC-AGI Leaderboard by asdfologist on Saturday July 25, 2026 @09:45AM (#49047546)
Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4).

It's more likely that the training data was contaminated with the benchmark data.

Re:ARC-AGI Leaderboard by mupuff1234 on Saturday July 25, 2026 @10:02AM (#49047656)
You really think they saw the jump in arc-agi-3 (which they reported in their official card), and didn't even bother to check?

They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .

Re:ARC-AGI Leaderboard by johnfn on Saturday July 25, 2026 @12:18PM (#49048884)
How do you propose they check for something like this? They can't exactly ctrl-f the model weights for "Arc-AGI".
Re:ARC-AGI Leaderboard by vickychijwani on Saturday July 25, 2026 @02:53PM (#49050455)
Anthropic expends tons of compute and effort on understanding internal model states [1]; this kind of thing is right up their alley.

[1]: Recent example: https://www.anthropic.com/research/global-workspace

Re:ARC-AGI Leaderboard by phoghed on Saturday July 25, 2026 @12:56PM (#49049252)
> they can’t possibly know or find out what was in the training data

doesn’t appear to be a very strong argument

Re:ARC-AGI Leaderboard by goodmythical on Saturday July 25, 2026 @04:39PM (#49051335)
Is it really all that different from how the models "know" anything else?

The best way to know that christmas trees are typically purchased in december isn't to access historical financial tables and run the entire analysis yourself but to just "know it" from the articles you read attesting to the fact.

If you've found data that is the answer to the question, be it a random user question or a benchmark, the softwares' goal is to produce the correct solution and it is cheaper to retrieve from storage than to compute.

It's the same with coding. The agent usually isn't really thinking about the problem from first principles, it's just giving you the answers it's already found in instances where someone else asked the same question.

It seems to me that if you're really testing for reasoning capability, just as if you were an instructor administering a test, you'll need to change the test from run to run in order to make sure the agent/student isn't just copying old tests.