News for hackers. Stuff that matters.

A comment on GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models. Parent comment

Re:GPT Neo: open-source GPT model, with pretrained 1.3B & 2.... by nl on Monday March 22, 2021 @12:38AM (#26536917)
Almost all the challenges with GPT-sized models are engineering and training challenges, not architectural.

How do you train a model too big to fit in a single GPU? It's doable, but not simple. How do you update weights across your cluster? etc etc

[ Parent | on HN ]

There are no comments on this comment yet.