Understanding Reasoning LLMs

Re:Understanding Reasoning LLMs by suddenlybananas on Thursday February 06, 2025 @07:00PM (#42967765)

>They came into this with the assumption that LLMs are just a cheap trick. As a result, they deliberately searched for an example of failure, rather than trying to do an honest assessment of generalization capabilities.

And lo and behold, they still found a glaring failure. You can't fault them for not buying into the hype.

[ Parent | on HN ]

Re:Understanding Reasoning LLMs by Legend2440 on Thursday February 06, 2025 @07:06PM (#42967805)

But it is still dishonest to declare reasoning LLMs a scam simply because you searched for a failure mode.

If given a few hundred tries, I bet I could find an example where you reason poorly too. Wikipedia has a whole list of common failure modes of human reasoning: https://en.wikipedia.org/wiki/List_of_fallacies

[ Parent | on HN ]

Re:Understanding Reasoning LLMs by daveguy on Thursday February 06, 2025 @07:44PM (#42968070)

Well, given the success rate is no more than 90% in the best cases. You could probably find a failure in about 10 tries. The only exception is o1-preview. And this is just a simple substitution of parameters.

[ Parent | on HN ]