TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

Re:TinyLlama project aims to pretrain a 1.1B Llama model on ... by jph00 on Monday September 04, 2023 @05:25PM (#37385230)

AFAIK re-warming it up and then gradually decreasing it again ought to work fine. Have you seen any research showing that it doesn't?

[ Parent | on HN ]

Re:TinyLlama project aims to pretrain a 1.1B Llama model on ... by fpgaminer on Monday September 04, 2023 @06:32PM (#37385759)

That would work, in that it would allow one to continue decreasing the loss, but I wouldn't say that it would work "fine". A model trained with restarts always performs worse than a model trained for the same duration without restarts.

[ Parent | on HN ]

Re:TinyLlama project aims to pretrain a 1.1B Llama model on ... by jph00 on Monday September 04, 2023 @08:16PM (#37386460)

I haven't seen any researcher that supports your contention. SGDR (SGD with restarts) has been shown to work well. https://arxiv.org/abs/1608.03983

[ Parent | on HN ]

Re:TinyLlama project aims to pretrain a 1.1B Llama model on ... by two_in_one on Monday September 04, 2023 @11:35PM (#37387557)

> A model trained with restarts always performs worse than a model trained for the same duration without restarts.

Citation would be nice. From my experience restart sometimes is required. When model gets unstable and 'explodes', or gets stuck in some local minima. This is common with GANs. I usually rollback the model a bit, but keep the latest discriminator. So that discriminator 'knows' what to expect. It works in most cases, except for the 'fatality', when model blows up no matter what. That's the end of training.

[ Parent | on HN ]