GPT-Neo – Building a GPT-3-sized model, open source and free

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by joshlk on Monday January 18, 2021 @06:13AM (#25820538)

How does the outfit intend to fund the project? OpenAI spends millions on computing resources to train the models.

[ Parent | on HN ]

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by stellaathena on Monday January 18, 2021 @10:54AM (#25822947)

Hey! One of the lead devs here. A cloud computing company called CoreWeave is giving us the compute for free in exchange for us releasing it. We're currently at the ~10B scale and are working on understanding datacenter scale parallelized training better, but we expect to train the model on 300-500 V100s for 4-6 months.

[ Parent | on HN ]

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by tmalsburg2 on Monday January 18, 2021 @06:28AM (#25820651)

I imagine recreating the model will be computationally cheaper because they will not have to sift through the same huge hyperparameter space as the initial GPT-3 team had to.

[ Parent | on HN ]

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by jne2356 on Monday January 18, 2021 @11:38AM (#25823486)

This is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361). The compute savings from not having to do the hyperparameter sweep are negligible and do not significantly change the feasibility of the project.

[ Parent | on HN ]

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by thelastwave on Monday January 18, 2021 @11:35AM (#25823440)

Why is that?

[ Parent | on HN ]

Re:GPT-Neo – Building a GPT-3-sized model, open source and free by jne2356 on Monday January 18, 2021 @11:35AM (#25823447)

The cloud company CoreWeave has agreed to provide the GPU resources necessary.

[ Parent | on HN ]