News for hackers. Stuff that matters.
Re:ARC-AGI Leaderboard by KaoruAoiShiho on Saturday July 25, 2026 @08:36AM (#49047099)
Re:ARC-AGI Leaderboard by jchw on Saturday July 25, 2026 @11:24AM (#49048358)
I have been claiming that I don't think Chinese AI companies are benchmaxxing harder than American AI companies, which has gotten mixed reception: sometimes people agree, sometimes they disagree.

It seems I was wrong. American AI companies might actually be benchmaxxing harder.

Re:ARC-AGI Leaderboard by gertlabs on Saturday July 25, 2026 @02:32PM (#49050269)
We run an evaluation that is designed to be less vulnerable to benchmaxxing because there aren't correct solutions; agents are interacting in the same environment as other agents. And it's private, and our public benchmark is not well known enough for anyone to probably care to benchmax us yet. So I think it's pretty indicative of true relative aptitude.

All models have probably memorized significant swaths of solution sets for popular benchmarks at this point, either accidentally or intentionally, so it's all relative at this point. However, in our experience, Chinese models do benchmax harder. This is also consistent with interacting with Chinese labs soliciting data/environments, who literally asked us for datasets and tasks modeled around and formatted like popular benchmarks.

Opus 5 will be uploaded tomorrow, but we already have the tests locally and it is truly as capable as Fable, but at 81% of the real cost. (And from subjective usage, it has a very different personality)

Data at https://gertlabs.com/rankings

Re:ARC-AGI Leaderboard by jchw on Saturday July 25, 2026 @03:22PM (#49050676)
Benchmaxxing via memorization is boring and doesn't fool anyone for too long. It works, but then new benchmarks test old models and the real results fall in line. Benchmaxxing by focusing on specific types of things that benchmarks test on, while still not improving intelligence or capability in the general case? Not only is it blatantly obvious that all AI labs do this, but it's not even obvious how you would go about it any other way.

Now I am not really specifically accusing Anthropic of anything here, I'm just saying their behavior is suspicious. Since you tested Fable, they wouldn't even have to lie to have optimized for your specific benchmarks, since they absolutely had permission to read your sessions if they wanted to. But obviously, that's only the situation if we take them at their word. Personally I would be a bit surprised if they just flat out were lying and secretly retaining data they say they are not, but not that surprised. The penalties for doing this are probably worth the rewards if it keeps them super far ahead in the benchmarks for years without anyone catching on.

(In actuality though, even if they really were trying to sneakily grab samples of benchmark tests via their Fable data retention rules, I don't really suspect there would've been very much time to optimize Opus 5 on it. So consider me bothered.)

Re:ARC-AGI Leaderboard by kimjune01 on Sunday July 26, 2026 @12:46AM (#49054816)
someone's promotion depends on benching harder
Re:ARC-AGI Leaderboard by throwa356262 on Saturday July 25, 2026 @08:43AM (#49047153)
"Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration."

I guess there is no way this can happen without benchmark being part of the training data??

Re:ARC-AGI Leaderboard by pierrefermat1 on Saturday July 25, 2026 @09:12AM (#49047332)
What seems to implied is that some of his hold out testing suite includes simple/common tests that are out in the wild, and for those opus went straight to a memorised solution .
Re:ARC-AGI Leaderboard by zamadatix on Saturday July 25, 2026 @10:16AM (#49047745)
Simple/common tests is not an explanation for why only now Opus 5 is the only model encoding the answers like this. Something like the holdout test suite being leaked or Anthropic cheating (e.g. 'accidentally' including previous hold out run data in Opus 5 training) makes a much stronger fit.
Re:ARC-AGI Leaderboard by stogot on Saturday July 25, 2026 @09:15AM (#49047345)
It may read information about the benchmark, such as on blog post, or Twitter feeds (example OP) without active cheat
Re:ARC-AGI Leaderboard by modeless on Saturday July 25, 2026 @02:16PM (#49050120)
They state the puzzle is "Witness-like" which I assume means that it follows the rules from the well-known puzzle game "The Witness" which Opus definitely knows.
Re:ARC-AGI Leaderboard by rbuccigrossi on Saturday July 25, 2026 @08:12PM (#49053210)
No, I believe you are misunderstanding the quote. Each “question” in ARC-AGI-3 is a game that has hidden rules that you can understand if you look at the game board long enough. This quote means that Opus 5 is looking at the game board, figuring out the rules, and writing out the rules before it makes a single move. You can do the same thing if you go to the ARC-AGI-3 website and try some of the games.
Re:ARC-AGI Leaderboard by jnwatson on Saturday July 25, 2026 @09:05AM (#49047289)
I was just thinking they need to mark each model per benchmark as "model released before the benchmark was released" and "model released after the benchmark was released".
Re:ARC-AGI Leaderboard by nozzlegear on Saturday July 25, 2026 @10:18AM (#49047762)
Re:ARC-AGI Leaderboard by andrepd on Saturday July 25, 2026 @09:19AM (#49047367)
I'm shocked, astonished even, that enterprises on which trillions of dollars are being poured would consider cheating on marketing benchmarks.
Re:ARC-AGI Leaderboard by asdfologist on Saturday July 25, 2026 @09:45AM (#49047546)
Meh, I doubt it was intentional. Deliberate benchmaxxing is incredibly damaging to credibility once it's discovered (see what happened to Meta with LlaMa 4).

It's more likely that the training data was contaminated with the benchmark data.

Re:ARC-AGI Leaderboard by mupuff1234 on Saturday July 25, 2026 @10:02AM (#49047656)
You really think they saw the jump in arc-agi-3 (which they reported in their official card), and didn't even bother to check?

They maybe have not intentionally benchmaxxed, but they certainly know that's what happened .

Re:ARC-AGI Leaderboard by johnfn on Saturday July 25, 2026 @12:18PM (#49048884)
How do you propose they check for something like this? They can't exactly ctrl-f the model weights for "Arc-AGI".
Re:ARC-AGI Leaderboard by vickychijwani on Saturday July 25, 2026 @02:53PM (#49050455)
Anthropic expends tons of compute and effort on understanding internal model states [1]; this kind of thing is right up their alley.

[1]: Recent example: https://www.anthropic.com/research/global-workspace

Re:ARC-AGI Leaderboard by phoghed on Saturday July 25, 2026 @12:56PM (#49049252)
> they can’t possibly know or find out what was in the training data

doesn’t appear to be a very strong argument

Re:ARC-AGI Leaderboard by goodmythical on Saturday July 25, 2026 @04:39PM (#49051335)
Is it really all that different from how the models "know" anything else?

The best way to know that christmas trees are typically purchased in december isn't to access historical financial tables and run the entire analysis yourself but to just "know it" from the articles you read attesting to the fact.

If you've found data that is the answer to the question, be it a random user question or a benchmark, the softwares' goal is to produce the correct solution and it is cheaper to retrieve from storage than to compute.

It's the same with coding. The agent usually isn't really thinking about the problem from first principles, it's just giving you the answers it's already found in instances where someone else asked the same question.

It seems to me that if you're really testing for reasoning capability, just as if you were an instructor administering a test, you'll need to change the test from run to run in order to make sure the agent/student isn't just copying old tests.

Re:ARC-AGI Leaderboard by root-parent on Saturday July 25, 2026 @12:24PM (#49048929)
And being worst than previous model...

"...The traces tell the why: (1) On our most classic Witness-style game, Opus 5 states the hidden rules before its first action, then plays a byte-identical optimal solution in 5/5 seeds at temperature 1.0. Zero exploration. It already knows this genre. (2) But on our most novel game (unusual mechanic combinations you can't pattern-match), Opus 5 regresses below Opus 4.8. Where rules must actually be discovered through interaction, the new model is worse than the old one..."

Re:ARC-AGI Leaderboard by Zababa on Saturday July 25, 2026 @12:29PM (#49048978)
>That decomposition (perfect on templates, regressed on novelty) is the signature of “scaffold-then-internalize” training on genre-specific data, not a general gain in interactive abstract reasoning.

They're smuggling a claim that benchmarks like ARC-AGI measure "interactive abstract reasoning" here, which is what is claimed by the people that make these benchmarks, and also not proven.

Re:ARC-AGI Leaderboard by FairyFair_ on Tuesday July 28, 2026 @07:33AM (#49082316)
Hello, sorry. I know this is not relevant to what you are talking about. However I have been unable to contact you by any other means and have been trying to contact you for more than a month Assuming that you are "kas", please check the dulst discord. It is important.