Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by corimaith on Thursday October 31, 2024 @02:51PM
(#42010153)
The reasoning emerges from the long distance relations between words picked up by the parallel nature of the transformers. It's why they were so much more performant than earlier RNNs and LSTMs which were using similar tokenization.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by cjbprime on Friday November 01, 2024 @12:10PM
(#42018399)
The argument isn't that there is something more than next token prediction happening.
The argument is that next token prediction does not imply an upper bound on intelligence, because an improved next token prediction will pull increasingly more of the world that is described in the training data into itself.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by unoti on Friday November 01, 2024 @03:24PM
(#42020591)
> The argument isn't that there is something more than next token prediction happening.
> The argument is that next token prediction does not imply an upper bound on intelligence, because an improved next token prediction will pull increasingly more of the world that is described in the training data into itself.
Well said! There's a philosophical rift appearing in the tech community over this issue semi-neatly dividing people between naysayers, "disbelievers" and believers over this very issue.
Re:Chain-of-Thought Can Hurt Performance on Tasks Where Thin...
by nuancebydefault on Friday November 01, 2024 @05:43PM
(#42021841)
I fully agree. Some people fully disagree though on the 'pull of the world' part, let alone 'intelligence' part, which are in fact impossible to define.
We don't know how reasoning emerges in humans. I'm pretty sure the multi-model-ness helps, but it is not needed for reasoning, because they imply other forms of input, hence just more (be it somewhat different) input. A blind person can still form an 'image'.
In the same sense, we don't know how reasoning emerges in LLMs. For me the evidence lays in the results, rather than in how it works. For me the results are enough of an evidence.