News for hackers. Stuff that matters.
Re:Achieving AI-style reasoning without an LLM by chrisjj on Thursday October 22, 2026 @07:05AM (#50193757)
> Next token prediction is still how pre-training works, but that's a relatively small part of the whole training run nowadays

Only if you call 90% small.

No-one is claiming <50%.

Re:Achieving AI-style reasoning without an LLM by cubefox on Thursday October 22, 2026 @08:25AM (#50194357)
It's only about 10% of total compute capacity. https://x.com/EugeneNg/status/2099315982959616369
Re:Achieving AI-style reasoning without an LLM by didroe on Thursday October 22, 2026 @09:41AM (#50195196)
I think there's a question of semantics here. RL is way more expensive in terms of compute, as you have to generate a significant amount of output before you can judge whether it's good or bad. So compute is one angle, but another is how much impact RL has on the weights vs pre-training
Re:Achieving AI-style reasoning without an LLM by chrisjj on Thursday October 22, 2026 @10:00AM (#50195439)
I cannot see how cost is relevent. What we're discussing is impact.
Re:Achieving AI-style reasoning without an LLM by cubefox on Thursday October 22, 2026 @11:41AM (#50196957)
Apparently the impact of RLVR is generally bigger, otherwise AI companies would spend more of their compute budget on pre-training. The former seems to be more effective.
Re:Achieving AI-style reasoning without an LLM by chrisjj on Thursday October 22, 2026 @01:44PM (#50198691)
That kinda presumes more pre-training compute would significantly help. There is a diminishing returns curve.