News for hackers. Stuff that matters.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by og_kalu on Monday October 30, 2023 @11:43PM (#38080015)
Prediction as an objective basically forces the models to model the casual processes that create the text itself. It's not going to stop getting better unless the data is insufficient/unvaried or the architecture creates a bottleneck.

I think by the time the former is an "issue", we'll have a Super Intelligence on our hands anyway.

The latter is looking less and less likely to be a real hurdle. Very little inductive bias to steer away from crucial solutions, very scalable.

There are no comments on this comment yet.