News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jprete on Wednesday January 01, 2025 @08:49AM (#42565995)
I don't see any reason to assume they removed it unless they're very explicit about it. Model publishers have an extremely strong vested interest in beating benchmarks and I expect them to teach to the test if they can get away with it.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @12:03PM (#42567309)
putnam isn’t an llm benchmark ahhhh none of these companies are reporting putnam scores there’s nothing nefarious about training on putnam problems
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jprete on Thursday January 02, 2025 @07:04AM (#42573709)
Any problem set that can make news is implicitly an LLM benchmark.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by stingraycharles on Wednesday January 01, 2025 @09:42AM (#42566263)
As usual, once a metric becomes a target, it stops being useful.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by franktankbank on Wednesday January 01, 2025 @11:45AM (#42567150)
Well, they are doing BigCorpStuff not Science