News for hackers. Stuff that matters.

A comment on TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens. Parent comment

Re:TinyLlama project aims to pretrain a 1.1B Llama model on ... by jph00 on Monday September 04, 2023 @08:16PM (#37386460)
I haven't seen any researcher that supports your contention. SGDR (SGD with restarts) has been shown to work well. https://arxiv.org/abs/1608.03983
[ Parent | on HN ]

There are no comments on this comment yet.