Re:How Meta trains large language models at scale
by uluyol on Thursday June 13, 2024 @01:20AM
(#40666205)
Google most certainly uses TPUs for training.
Re:How Meta trains large language models at scale
by refulgentis on Thursday June 13, 2024 @01:20AM
(#40666197)
This breaks my brain, because I know Google trains it models on TPUs and they're seen as faster, and if they're better at inference, and can train, then why is Nvidia in a unique position? My understanding was always it's as simple as it required esoteric tooling
Re:How Meta trains large language models at scale
by smueller1234 on Thursday June 13, 2024 @01:25AM
(#40666236)
Multiple types of TPUs.
(I work for Google, but the above is public information.)
Re:How Meta trains large language models at scale
by vineyardmike on Thursday June 13, 2024 @11:24AM
(#40670808)
Because people generally don’t use TPUs outside of Google. The tooling is different, the access is metered through GCP, etc.
Nvidia is in a vaguely unique position in that their products have great tooling support and few companies sell silicon at their scale.
Re:How Meta trains large language models at scale
by refulgentis on Thursday June 13, 2024 @01:07PM
(#40672092)
Correct, I'm pointing out politely that's in conflict with the person I'm replying to.
They do sell them - but through their struggling cloud business. Either way, Nvidia's margin is google's opportunity to lower costs.
> I can see Google's custom chips are 15x to 30x slower to train AI
TPUs are designed for inference not training - they're betting that they can serve models to the world at a lower cost structure than their competition. The compute required for inference to serve their billions of customers is far greater than training costs for models - even LLMs. They've been running model inference as a part of production traffic for years.