News for hackers. Stuff that matters.
Re:Stability AI Launches the First of Its StableLM Suite of ... by MacsHeadroom on Wednesday April 19, 2023 @07:48PM (#35634723)
The optimal training tokens for 65B parameters is like 80T.

Emad tweeted "Goin to train a 3B model on 3T tokens" last month. These 800B checkpoints are just early alpha training checkpoints.

The full training set is 1.5T currently and will likely grow.

There are no comments on this comment yet.