Re:Achieving AI-style reasoning without an LLM
by chrisjj on Thursday October 22, 2026 @07:05AM
(#50193757)
> Next token prediction is still how pre-training works, but that's a relatively small part of the whole training run nowadays
Only if you call 90% small.
No-one is claiming <50%.
Re:Achieving AI-style reasoning without an LLM
by cubefox on Thursday October 22, 2026 @08:25AM
(#50194357)
It's only about 10% of total compute capacity.
https://x.com/EugeneNg/status/2099315982959616369
Re:Achieving AI-style reasoning without an LLM
by didroe on Thursday October 22, 2026 @09:41AM
(#50195196)
I think there's a question of semantics here. RL is way more expensive in terms of compute, as you have to generate a significant amount of output before you can judge whether it's good or bad. So compute is one angle, but another is how much impact RL has on the weights vs pre-training
Re:Achieving AI-style reasoning without an LLM
by chrisjj on Thursday October 22, 2026 @10:00AM
(#50195439)
I cannot see how cost is relevent. What we're discussing is impact.
Re:Achieving AI-style reasoning without an LLM
by cubefox on Thursday October 22, 2026 @11:41AM
(#50196957)
Apparently the impact of RLVR is generally bigger, otherwise AI companies would spend more of their compute budget on pre-training. The former seems to be more effective.
Re:Achieving AI-style reasoning without an LLM
by chrisjj on Thursday October 22, 2026 @01:44PM
(#50198691)
That kinda presumes more pre-training compute would significantly help. There is a diminishing returns curve.
Re:Achieving AI-style reasoning without an LLM
by asdf88990 on Thursday October 22, 2026 @08:54AM
(#50194675)
All that happens in reinforcement learning and everything from pre-training to generation, including the last token, at the end is about adjusting the probability of the next token given the sequence.
Anyone who thinks reinforcement learning changes how the generation works hasn’t worked on inference engines and you can dismiss them as inexperienced and misguided.
Generally, anyone talking about next token prediction being some limiting factor of LLM capability is a good reason to ignore them, as they haven't updated their understanding of LLMs since 2023 or so.