News for hackers. Stuff that matters.

A comment on 30% Drop In o1-Preview Accuracy When Putnam Problems Are Slightly Variated. Parent comment

Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by 1209412comb on Wednesday January 01, 2025 @11:16PM (#42571591)
I tried 3 times the "Which is heavier, a 10.01kg block of steel on or a 9.99kg bag of feathers?" and ChatGPT keep converting kg to pound and saying the 9.99kg is heavier.
[ Parent | on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by fragmede on Thursday January 02, 2025 @07:11AM (#42573749)
Which model? On the paid plus tier, GPT-4o, GPT-o1, and GPT-o1mini all successfully got the 10.1. I did not try any other models.

gpt-4o: https://chatgpt.com/share/67768221-6c60-8009-9988-671beadb5a...

o1-mini: https://chatgpt.com/share/67768231-6490-8009-89a6-f758f0116c...

o1: https://chatgpt.com/share/67768254-1280-8009-aac9-1a3b75ccb4...

[ Parent | on HN ]
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ... by mycall on Thursday January 02, 2025 @12:15PM (#42576308)
Did you try it more than once per model? I could see temperature or other similar gradients providing wrong answers.
[ Parent | on HN ]