News for hackers. Stuff that matters.

A comment on Facebook LLAMA is being openly distributed via torrents. Parent comment

Re:Facebook LLAMA is being openly distributed via torrents by krisoft on Friday March 03, 2023 @08:30AM (#35009321)
> No you are probably overestimating the cost by 1-2 orders of magnitude.

You are right! Wow. Thank you for correcting me.

> GPT-3 probably cost under $5 million,

Is that one training run or includes all the fiddling to find the right hyperparameters? Or there aren't many of those in these training or they are not that sensitive?

[ Parent | on HN ]
Re:Facebook LLAMA is being openly distributed via torrents by sebzim4500 on Friday March 03, 2023 @10:01AM (#35010456)
I think they probably did a lot of hyperparameter searching to train the smaller models and then extrapolated for the largest model, but I'm just guessing. OpenAI had a finite amount of money when they were training GPT-3, they likely do it differently now that inference costs are significant compared to training costs.
[ Parent | on HN ]