News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by e1g on Wednesday January 01, 2025 @09:34AM (#42566224)
The paper includes several examples of their modified questions. There has been a substantial jump from o1-preview to o1, so I gave several samples to o1 and o1-pro (not o1-preview), and current o1s gave the correct answer to those modified problems. SOTA changes fast.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jtefera on Wednesday January 01, 2025 @10:17AM (#42566496)
The paper mentions that on several occasions the LLM will provide a correct answer but will either take big jumps without justifying them or will take illogical steps but end up with the right solution at the end. Did you check for that?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by e1g on Wednesday January 01, 2025 @10:24AM (#42566544)
No, I don't know enough math to test the logic, only the check questions against their expected answers in https://anonymous.4open.science/r/putnam-axiom-B57C/data/Put...
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zeroonetwothree on Wednesday January 01, 2025 @10:54AM (#42566752)
Putnam problems need to actually be graded, often the answer itself is trivial.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by suddenlybananas on Wednesday January 01, 2025 @09:43AM (#42566267)
LLM boosters are so tiresome. You hardly did a rigorous evaluation, the set has been public since October and could have easily been added to the training data.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by e1g on Wednesday January 01, 2025 @10:03AM (#42566396)
I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check.

Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test.

Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by fumeux_fume on Wednesday January 01, 2025 @11:23AM (#42566971)
> "...roughly on par with a mediocre, but not completely incompetent graduate student"

Let it sink in how vague and almost meaningless that statement is.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by pizza on Wednesday January 01, 2025 @05:32PM (#42569862)
What types of questions are you hoping to answer for that to be considered a vague statement?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by gdhkgdhkvff on Wednesday January 01, 2025 @09:55AM (#42566340)
Your points would be more convincing if you didn’t preface them with arrogant cynicism.