Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by suddenlybananas on Wednesday January 01, 2025 @09:43AM
(#42566267)
LLM boosters are so tiresome. You hardly did a rigorous evaluation, the set has been public since October and could have easily been added to the training data.
[ Parent
| on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by e1g on Wednesday January 01, 2025 @10:03AM
(#42566396)
I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check.
Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test.
Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by fumeux_fume on Wednesday January 01, 2025 @11:23AM
(#42566971)
> "...roughly on par with a mediocre, but not completely incompetent graduate student"
Let it sink in how vague and almost meaningless that statement is.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by pizza on Wednesday January 01, 2025 @05:32PM
(#42569862)
What types of questions are you hoping to answer for that to be considered a vague statement?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by gdhkgdhkvff on Wednesday January 01, 2025 @09:55AM
(#42566340)
Your points would be more convincing if you didn’t preface them with arrogant cynicism.