jne2356

The resources for the GPT-3 replication will be provided by the cloud company CoreWeave.

They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible.

https://github.com/EleutherAI/info

OpenAI only trained the full sized GPT-3 once. Hyperparameter sweep was conducted on significantly smaller models (see: https://arxiv.org/abs/2001.08361)

This is not true. The OpenAI team only trained one full-sized GPT-3, and conducted their hyperparameter sweep on significantly smaller models (see: https://arxiv.org/abs/2001.08361). The compute savings from not having to do the hyperparameter sweep are negligible and do not significantly change the feasibility of the project.

EleutherAI has already secured the resources necessary.

They get the seti@home suggestion a lot. There's a section in their FAQ that explains why it's infeasible.

https://github.com/EleutherAI/info

The cloud company CoreWeave has agreed to provide the GPU resources necessary.

GPT-3 does not fit in any one GPU that exists at present. It's already spread out across multiple GPUs.

They get this suggestion a lot. There's a section in their FAQ that explains why it's infeasible.

https://github.com/EleutherAI/info