Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by wongarsu on Wednesday January 01, 2025 @04:22PM
(#42569422)
We haven't coded LLMs to be stochastic models, we coded them to predict text with any method gradient decent finds on a transformer architecture. That's not exactly the same.
But more importantly, if you want to show that LLMs can't reason you obviously have to use a test that when applied to humans would show that humans can reason. Otherwise your test isn't testing reasoning but something more strict.
Re:30% Drop In o1-Preview Accuracy When Putnam Problems Are ...
by Isinlor on Wednesday January 01, 2025 @05:30PM
(#42569851)
It's widely accepted that reasoning is not a binary skill.
You can make mistakes and still reason. Very often people given the same premises will disagree in thier reasoning as we are doing right here.
There's deeper philosophical questions about what reasoning actually _is_, and LLMs have made those sharper, because they've shown it's clearly possible for a complex statistical model to generate words that look like reasoning, but the question is whether there's a difference between what they're doing and what humans are doing, and evidence that they're _not_ reasoning - evidence that they're just generating words in specific orders - weighs heavily against them.