Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by captainbland on Wednesday January 01, 2025 @09:01AM
(#42566058)
I think it's reasonable to assume that openAI is optimising for maximum hype at this point which may include wilfully overfitting for impactful benchmarks to generate positive reports.
[ Parent
| on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by lupire on Wednesday January 01, 2025 @09:06AM
(#42566089)
When 4 came out they released a document that did BOTH inflate scores by changing the exam conditions, and also bragged about scoring worse than guessing on a multiple choice test.