News for hackers. Stuff that matters.
Re:How We Broke Top AI Agent Benchmarks: And What Comes Next by stingraycharles on Sunday April 12, 2026 @01:09AM (#47736305)
This is already well known, all these AI benchmarks use a different model to judge whether or not the solution was correct.

It’s… remarkably poor, and as demonstrated in the paper, easily gamed. Worst yet, these benchmarks teach AIs to be very short-sighted and hyper-focused on completing the task, rather than figuring out the best solution.

There are no comments on this comment yet.