News for hackers. Stuff that matters.
Re:Gemini 3 Pro Preview Live in AI Studio by rafaelmn on Tuesday November 18, 2025 @11:50PM (#45975977)
It tells me that the benchmark is probably leaking into training data, and going to the benchmark site :

> Model was published after the competition date, making contamination possible.

Aside from eval on most of these benchmarks being stupid most of the time, these guys have every incentive to cheat - these aren't some academic AI labs, they have to justify hundreds of billions being spent/allocated in the market.

Actually trying the model on a few of my daily tasks and reading the reasoning traces all I'm seeing is same old, same old - Claude is still better at "getting" the problem.

There are no comments on this comment yet.