Re:Who uses Google TPUs for inference in production?
by hiddencost on Monday March 11, 2024 @02:40PM
(#39671847)
Google is using them in prod. I think they're so hungry for chips internally that cloud isn't getting much support in selling them.
[ Parent
| on HN ]
Re:Who uses Google TPUs for inference in production?
by _b on Monday March 11, 2024 @04:29PM
(#39672910)
I think this is right, in part because I've been told exactly this from people who work for Google and their job is to sell me cloud stuff- i.e., they say they have so much internal demand they aren't pushing TPUs for external use. Hence external pricing and support just isn't that great right now. But presumably when capacity catches up they'll start pushing TPUs again.
Re:Who uses Google TPUs for inference in production?
by natbobc on Monday March 11, 2024 @06:08PM
(#39673799)
Feels like a bad point in the curve to try and sell them. “Oh our internal hypecycle is done… we’ll put them in the market now that they’re all worn out.
Re:Who uses Google TPUs for inference in production?
by rj45jackattack on Monday March 11, 2024 @06:48PM
(#39674098)
Sounds like ButterflyLabs.
Re:Who uses Google TPUs for inference in production?
by danjl on Monday March 11, 2024 @03:51PM
(#39672521)
I would guess that Google's vertexAI managed solution uses TPUs. Also Google uses them internally to train and infer for all their research products.
Re:Who uses Google TPUs for inference in production?
by michaelt on Monday March 11, 2024 @03:56PM
(#39672585)
While you can use TPUs with vertexai, it's just virtual machines - you can have one with an nvidia card if you like.
Re:Who uses Google TPUs for inference in production?
by sciencesama on Monday March 11, 2024 @03:51PM
(#39672538)
80 to 90% are consumed internally !! Only from version 5 it is planned to be customer focussed !!
Re:Who uses Google TPUs for inference in production?
by VirusNewbie on Monday March 11, 2024 @04:12PM
(#39672738)
They're getting swallowed up by Anthropic and the other huge spenders:
https://www.prnewswire.com/news-releases/google-announces-ex...
"Partnership includes important new collaborations on AI safety standards, committing to the highest standards of AI security, and use of TPU v5e accelerators for AI inference "
Re:Who uses Google TPUs for inference in production?
by amelius on Monday March 11, 2024 @03:16PM
(#39672212)
Or maybe they are just using nVidia. Who knows ...
Re:Who uses Google TPUs for inference in production?
by vineyardmike on Monday March 11, 2024 @03:51PM
(#39672526)
Beyond the fact that this is hardly a secret, there’s lots of other signs.
1. They have bought far less from NVidia than other hyper scalers, and they literally can’t vomit without saying “AI”. They have to be running those models on something. They have purchased huge amounts of chips from fabs, and what else would that be?
2. They have said they use them. Should be pretty obvious here.
3. They maintain a whole software stack for them, they design the chips, etc. Then they don’t really try to sell the TPU. Why else would they do this?
Re:Who uses Google TPUs for inference in production?
by jeffbee on Monday March 11, 2024 @05:06PM
(#39673231)
They have announced publicly using TPUs for inference, as far back as 2016. They did not offer TPUs for Cloud customers until 2017. The development is clearly driven by internal use cases. One of the features they publicly disclosed as TPU-based was Smart Reply and that launched in 2015. So their internal use of TPUs for inference goes back nearly a decade.