News for hackers. Stuff that matters.

A comment on GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models. Parent comment

Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by Voloskaya on Sunday March 21, 2021 @07:26PM (#26535088)
GPT-3 has 175 billion parameters. So they need to scale by 64x. They already have a comparable amount of data than what was used by OpenAI, so it's about scaling the numbers of GPUs.
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by f430 on Sunday March 21, 2021 @07:52PM (#26535239)
I see so that means this GPT Neo is 64 less powerful?
[ Parent | on HN ]
Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by Voloskaya on Sunday March 21, 2021 @09:32PM (#26535875)
Accuracy and numbers of parameters don't scale linearly together. It varies widely depending on exactly what you are measuring accuracy on as well etc. But a very approximate rule of thumb would be to say that accuracy scales with the log of the parameter count (for the same architecture).
[ Parent | on HN ]