News for hackers. Stuff that matters.
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by PhilippGille on Saturday December 21, 2024 @05:18AM (#42478738)
Benchmark fraud is not a novel concept. Outside of LLMs for example smartphone manufacturers detect benchmarks and disable or reduce CPU throttling: https://www.theregister.com/2019/09/30/samsung_benchmarking_...
Re:OpenAI O3 breakthrough high score on ARC-AGI-PUB by hmottestad on Saturday December 21, 2024 @08:58AM (#42479704)
CPU frequency ramp curve is also something that can be adjusted. You want the CPU to ramp up really quickly to make everything feel responsive, but at the same time you want to not have to use so much power from your battery.

If you detect that a benchmark is running then you can just ramp up to max frequency immediately. It’ll show how fast your CPU is, but won’t be representative of the actual performance that users will get from their device.