Microsoft (using Azure DCs) built a supercomputer with 10,000 V100 GPUs exclusively for OpenAI. [0]
It is estimated that it cost around $5M in compute time to train GPT-3.
OpenAI has received billions in investment prior to launching GPT-3, including $1B from Microsoft in 2019.
[0]: https://blogs.microsoft.com/ai/openai-azure-supercomputer/
Re:NanoGPT
by nileshtrivedi on Wednesday January 11, 2023 @05:25AM
(#34336968)
> we're not talking of Google or Facebook investing a chunk of their billions cash
OpenAI had raised $1B from Microsoft in 2019 and used it to train a 175B param model. Now, they have raised $10B and are training GPT-4 with 1.5T params. GPUs are capital intensive and as long as there are returns to bigger models, that's exactly where things will go.
I can't find any source on the 1.5T params number. I'd love to read more if you have any links to share. Thanks
afaik, gpt-4 is mostly rumours so far, same thing for the 1.5T number. gpt-4 is suerly coming.
Maybe it will be called GPT-XP by then, with Microsoft owning half of it.
Or GPT-365
Then they can bring back the talking paperclip, but this time actually useful.
Re:NanoGPT
by generalizations on Wednesday January 11, 2023 @10:47AM
(#34339737)
It could actually work. It would be an incredibly gutsy move and I love it, and they'd probably earn a lot of respect. They’d get so much press for it. And if it held up, it’d probably be one of the things that MS is remembered for.
Or GPT One.
Why not ask GPT itself what it wants to be called?
Looking forward to see GPT-4 recommending Linux and Libre Office instead of Windows/Office as the logical choice out of 250 IQ ML Model...
GPT-10 will be evergreen and 'the last version of GPT'.
And then three years later GPT-11 will be required to run the latest games.
In my imagination, OpenAI does what Bungie did when MS bought them, and open-sources what used to be their crown jewels.
That said, GPT-AlephOne only makes sense if there's a preceding GPT-∞.
They have got to release GPT-3.11 For Workgroups first.
Will 1.5T parameters be possible to run in the public way GPT-3 is? I can’t wait to see what happens with this much learning!
I am actually still unclear how AlephAlpha pulled that off and who funds them, since they have a rather low profile team.
Re:NanoGPT
by hdjjhhvvhga on Wednesday January 11, 2023 @06:19AM
(#34337267)
> we're not talking of Google or Facebook investing a chunk of their billions cash.
On the contrary, in this thread we are are mainly talking about that.
Re:NanoGPT
by SilverBirch on Wednesday January 11, 2023 @07:25AM
(#34337624)
OpenAI was founded in 2015 by a group of billionaires who pledged $1Bn of funding. That is hardly a small scrappy start up.
Sure, a single researcher can't replicate this at their university, but even though OpenAI likes to publish it this way, we're not really talking about research here. Research was inventing the transformer architecture, this is just making it bigger by (very smart) engineering choices. It's something companies should do (and are doing), not researchers.