News for hackers. Stuff that matters.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by suddenlybananas on Wednesday January 01, 2025 @09:43AM (#42566267)
LLM boosters are so tiresome. You hardly did a rigorous evaluation, the set has been public since October and could have easily been added to the training data.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by e1g on Wednesday January 01, 2025 @10:03AM (#42566396)
I'm not skilled enough in math to do a rigorous evaluation, so it was a quick check.

Terence Tao is skilled enough, and he describes O1's math ability is "...roughly on par with a mediocre, but not completely incompetent graduate student" (good discussion at https://news.ycombinator.com/item?id=41540902), and the next iteration O3 just got 25% on his brand new Frontier Math test.

Seeing LLMs as useless is banal, but downplaying their rate of improvement is self-sabotage.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by fumeux_fume on Wednesday January 01, 2025 @11:23AM (#42566971)
> "...roughly on par with a mediocre, but not completely incompetent graduate student"

Let it sink in how vague and almost meaningless that statement is.

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by pizza on Wednesday January 01, 2025 @05:32PM (#42569862)
What types of questions are you hoping to answer for that to be considered a vague statement?
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by gdhkgdhkvff on Wednesday January 01, 2025 @09:55AM (#42566340)
Your points would be more convincing if you didn’t preface them with arrogant cynicism.