Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by bee_rider on Wednesday January 01, 2025 @04:04PM
(#42569280)
Can’t really do science with closed source software, right? Who knows what’s in there.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by kbelder on Thursday January 02, 2025 @01:01PM
(#42576842)
I asked "Which is louder, a 12.5 decibel scream, or a 12.8 decibel whisper?", which I wouldn't think has much of a presence in its training set, and it answered correctly.
May still get it wrong in more subtle ways, though. Personally, I think it'll continue to get physics wrong until someone builds it some robot arms so it can train on actually interactive physical spaces and behavior.
In fact, in my use of o1 it's incredibly clear that it still has the same problems. It's incredibly common that the second I ask for someone even slightly outside the training set, it's more likely to "round" to some wrong solution in the training set, rather than use any sort of human-like reasoning to figure out the right answer (often the right answer isn't hard to get, just not found in a Google search).