News for hackers. Stuff that matters.

A comment on 30% Drop In o1-Preview Accuracy When Putnam Problems Are Slightly Variated. Parent comment

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by whimsicalism on Wednesday January 01, 2025 @12:03PM (#42567309)
putnam isn’t an llm benchmark ahhhh none of these companies are reporting putnam scores there’s nothing nefarious about training on putnam problems
[ Parent | on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jprete on Thursday January 02, 2025 @07:04AM (#42573709)
Any problem set that can make news is implicitly an LLM benchmark.
[ Parent | on HN ]