Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by amilios on Monday October 30, 2023 @10:35PM
(#38079502)
Loss was still decreasing for the models, there's a sense that we can push the training data much much further than we currently are.
[ Parent
| on HN ]
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by npsomaratna on Monday October 30, 2023 @11:12PM
(#38079807)
Yup. I found this article quite enlightening:
https://espadrine.github.io/blog/posts/chinchilla-s-death.ht...
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by rushingcreek on Tuesday October 31, 2023 @12:36AM
(#38080378)
Phenomenal blog post about scaling laws.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by og_kalu on Monday October 30, 2023 @11:43PM
(#38080015)
Prediction as an objective basically forces the models to model the casual processes that create the text itself. It's not going to stop getting better unless the data is insufficient/unvaried or the architecture creates a bottleneck.
I think by the time the former is an "issue", we'll have a Super Intelligence on our hands anyway.
The latter is looking less and less likely to be a real hurdle. Very little inductive bias to steer away from crucial solutions, very scalable.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs
by yorwba on Tuesday October 31, 2023 @03:15AM
(#38081205)
The TinyLlama project is trying to do that pushing by training a small 1.1 billion-parameter model on 3 trillion tokens: https://github.com/jzhang38/TinyLlama