News for hackers. Stuff that matters.

A comment on 30% Drop In o1-Preview Accuracy When Putnam Problems Are Slightly Variated. Parent comment

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by jtefera on Wednesday January 01, 2025 @10:17AM (#42566496)
The paper mentions that on several occasions the LLM will provide a correct answer but will either take big jumps without justifying them or will take illogical steps but end up with the right solution at the end. Did you check for that?
[ Parent | on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by e1g on Wednesday January 01, 2025 @10:24AM (#42566544)
No, I don't know enough math to test the logic, only the check questions against their expected answers in https://anonymous.4open.science/r/putnam-axiom-B57C/data/Put...
[ Parent | on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by zeroonetwothree on Wednesday January 01, 2025 @10:54AM (#42566752)
Putnam problems need to actually be graded, often the answer itself is trivial.
[ Parent | on HN ]