Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by og_kalu on Monday October 30, 2023 @11:43PM
(#38080015)
Prediction as an objective basically forces the models to model the casual processes that create the text itself. It's not going to stop getting better unless the data is insufficient/unvaried or the architecture creates a bottleneck.
I think by the time the former is an "issue", we'll have a Super Intelligence on our hands anyway.
The latter is looking less and less likely to be a real hurdle. Very little inductive bias to steer away from crucial solutions, very scalable.
I think by the time the former is an "issue", we'll have a Super Intelligence on our hands anyway.
The latter is looking less and less likely to be a real hurdle. Very little inductive bias to steer away from crucial solutions, very scalable.