Re:How Chain-of-Thought Reasoning Helps Neural Networks Compute
by sasja on Friday March 22, 2024 @11:04AM
(#39791426)
If you work out the loss function next token prediction, next 2 token prediction or next n token prediction, you will find they are identical. So it's equally correct to say the model is trained to find the most probable unlimited continuation. Saying "it only predicts the next token" is not untrue but easily leads to wrong conclusions.
[ Parent
| on HN ]
Re:How Chain-of-Thought Reasoning Helps Neural Networks Compute
by naasking on Friday March 22, 2024 @02:41PM
(#39793485)
> Saying "it only predicts the next token" is not untrue but easily leads to wrong conclusions.
Indeed, it's akin to saying that "only quantum fields exist" and then concluding that therefore people do not exist.