News for hackers. Stuff that matters.

A comment on Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models. Parent comment

Re:Cerebras-GPT: A Family of Open, Compute-Efficient, Large ... by option on Tuesday March 28, 2023 @03:08PM (#35346255)
it lags behind because according to their blogpost it was trained on <300B tokens. LLaMAs as far as I know were trained on more than trillion
[ Parent | on HN ]
Re:Cerebras-GPT: A Family of Open, Compute-Efficient, Large ... by gpm on Tuesday March 28, 2023 @03:41PM (#35346708)
The LLaMa paper says 1 trillion for the smaller models (7B, 13B) and 1.4 trillion for the larger models (30B, 65B)
[ Parent | on HN ]