News for hackers. Stuff that matters.
Re:How We Broke Top AI Agent Benchmarks: And What Comes Next by bee_rider on Saturday April 11, 2026 @04:51PM (#47733931)
What was the cheat in the 2024 Intel situation? The TomsHardware article and the Phoronix article they linked were quite vague. (Not to say I have any doubts, just curious, hadn’t heard of this one).
Re:How We Broke Top AI Agent Benchmarks: And What Comes Next by BugsJustFindMe on Saturday April 11, 2026 @07:37PM (#47734910)
Intel basically benchmaxxed their compiler optimizations. They used detailed knowledge of the benchmark to make their compiler generate machine code to do better on the benchmark in a way that was not beneficial for non-benchmark scenarios.
Re:How We Broke Top AI Agent Benchmarks: And What Comes Next by bee_rider on Sunday April 12, 2026 @02:07AM (#47736600)
I assumed as much, I’m just wondering what exactly they did. For example IIRC some phone company would detect that a benchmark was running by checking for the program name, and then allow the clock to boost higher (increase thermal limits) if it was a benchmark (like you could literally avoid the cheating behavior by changing the name of the program being run).