News for hackers. Stuff that matters.

A comment on The genie escapes: Stanford copies the ChatGPT AI for less than $600. Parent comment

Re:The genie escapes: Stanford copies the ChatGPT AI for les... by superkuh on Monday March 20, 2023 @05:10PM (#35238781)
The emerging consensus for larger LLM is you want to train them with at least 2-4x the tokens of the number of parameters (weights between neurons in the layers). A trillion (100x) surprises me.
[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by sebzim4500 on Monday March 20, 2023 @06:38PM (#35239847)
They probably put most of the effort into the 65B model, the 7B model was just trained so they could get an idea of the scaling behaviour. It makes sense to use the same amount of training steps, then.
[ Parent | on HN ]
Re:The genie escapes: Stanford copies the ChatGPT AI for les... by sitic on Monday March 20, 2023 @06:54PM (#35240015)
The LLaMA paper contradicts this view: "[...] Although Hoffmann et al. (2022) recommends training a 10B model on 200B tokens, we find that the performance of a 7B model continues to improve even after 1T tokens." https://arxiv.org/pdf/2302.13971.pdf
[ Parent | on HN ]