Re:Killed by LLM
by lxgr on Sunday January 05, 2025 @08:40PM
(#42606698)
Wasn’t that for human tests, i.e. not specifically AI benchmarks? Benchmarks should generally not be game-able by overfitting.
[ Parent
| on HN ]
The article shows all the tests against human performance.
The math one in particular is the one where small variations reduce the success rate significantly. I can’t find the source but it was pasted here in the last 2 weeks.
You're probably remembering this: https://news.ycombinator.com/item?id=42565606