Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by jprete on Wednesday January 01, 2025 @08:49AM
(#42565995)
I don't see any reason to assume they removed it unless they're very explicit about it. Model publishers have an extremely strong vested interest in beating benchmarks and I expect them to teach to the test if they can get away with it.
[ Parent
| on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by whimsicalism on Wednesday January 01, 2025 @12:03PM
(#42567309)
putnam isn’t an llm benchmark ahhhh none of these companies are reporting putnam scores there’s nothing nefarious about training on putnam problems
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by jprete on Thursday January 02, 2025 @07:04AM
(#42573709)
Any problem set that can make news is implicitly an LLM benchmark.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by stingraycharles on Wednesday January 01, 2025 @09:42AM
(#42566263)
As usual, once a metric becomes a target, it stops being useful.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by franktankbank on Wednesday January 01, 2025 @11:45AM
(#42567150)
Well, they are doing BigCorpStuff not Science