News for hackers. Stuff that matters.

A comment on GPT-Neo – Building a GPT-3-sized model, open source and free. Parent comment

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by emteycz on Monday January 18, 2021 @06:32AM (#25820677)
What about some kind of sharding, parts of the computation that could be executed in isolation for a longer period of time?
[ Parent | on HN ]
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by Filligree on Monday January 18, 2021 @09:00AM (#25821838)
An ongoing research problem. OpenAI would certainly like being able to use smaller GPUs, instead of having to fit the entire model into one.
[ Parent | on HN ]
Re:GPT-Neo – Building a GPT-3-sized model, open source and free by jne2356 on Monday January 18, 2021 @11:32AM (#25823400)
GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.
[ Parent | on HN ]