News for hackers. Stuff that matters.
Re:How We Broke Top AI Agent Benchmarks: And What Comes Next by robot-wrangler on Saturday April 11, 2026 @08:13PM (#47735075)
> evaluation was not designed to resist a system that optimizes for the score rather than the task.

Welcome to benchmarks in general, but especially reasoning. Robustness and sensitivity research says nothing is robust, everything is sensitive, feels like every paper says "yeah we made a new benchmark that shuffles the order of multiple choice options in the question set and found a 40% drop in model performance"

There are no comments on this comment yet.