30% Drop In o1-Preview Accuracy When Putnam Problems Are Slightly Variated

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by e1g on Wednesday January 01, 2025 @10:03AM (#42566396)

I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check.

Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test.

Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.

[ Parent | on HN ]

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by fumeux_fume on Wednesday January 01, 2025 @11:23AM (#42566971)

> "...roughly on par with a mediocre, but not completely incompetent graduate student"

Let it sink in how vague and almost meaningless that statement is.

[ Parent | on HN ]

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by pizza on Wednesday January 01, 2025 @05:32PM (#42569862)

What types of questions are you hoping to answer for that to be considered a vague statement?

[ Parent | on HN ]