News for hackers. Stuff that matters.

A comment on GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models. Parent comment

Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by f430 on Sunday March 21, 2021 @05:49PM (#26534434)
what does he mean when he says 1T or bust? Is he referring to 1 trillion parameters? Are you saying that GTP-3 has 2.7 trillion parameters? Does it mean that to get to GPT-3 level it needs 100x more amount of dataset?
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by jon_tau on Sunday March 21, 2021 @06:01PM (#26534537)
The saying comes from a slide by Noam Shazeer (see: https://www.youtube.com/watch?v=HgGyWS40g-g&ab_channel=Tenso...). It just means the current goal should be to have models with 1 trillion parameters.
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by pjfin123 on Sunday March 21, 2021 @06:00PM (#26534527)
I interpreted it as aspiring to a trillion paramters but I'm not sure.
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by Voloskaya on Sunday March 21, 2021 @07:26PM (#26535088)
GPT-3 has 175 billion parameters. So they need to scale by 64x. They already have a comparable amount of data than what was used by OpenAI, so it's about scaling the numbers of GPUs.
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by f430 on Sunday March 21, 2021 @07:52PM (#26535239)
I see so that means this GPT Neo is 64 less powerful?
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by Voloskaya on Sunday March 21, 2021 @09:32PM (#26535875)
Accuracy and numbers of parameters don't scale linearly together. It varies widely depending on exactly what you are measuring accuracy on as well etc. But a very approximate rule of thumb would be to say that accuracy scales with the log of the parameter count (for the same architecture).
[ Parent | on HN ]