Who uses Google TPUs for inference in production?

Re:Who uses Google TPUs for inference in production? by htrp on Monday March 11, 2024 @03:26PM (#39672298)

We've previously tried and almost always regretted the decision. I think the tech stack needs another 12-18 months to mature (doesn't help that almost all work ex Google is being done in torch).

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by mike_d on Monday March 11, 2024 @04:16PM (#39672771)

> I think the tech stack needs another 12-18 months to mature

Google has been doing AI before any other company even thought about it. They are on the 6th generation of TPU hardware.

I don't think there is any maturity issue, just an availability issue because they are all being used internally.

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by htrp on Monday March 11, 2024 @04:24PM (#39672845)

100% agree, if I have access to the TPU team internally, it'll be very easy to use in production.

If you aren't internal, the documentation, support, and even just general bug fixing is impossible to get.

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by ethbr1 on Monday March 11, 2024 @06:22PM (#39673902)

(Has an expert team dedicated solely to optimizing for exotic hardware) = an option

(Doesn't have a team like that) = stick to mass-use, commodity hardware

That's generally been the trade-off since ~1970. And usually, the performance isn't worth the people-salaries.

How many examples of successful hardware that isn't well-documented and doesn't have drop-in 1:1 SDK coverage vs (more popular solution) are there?

It seems like a heavy-lift to even get something that does have parity in those ways adopted, given you're fighting market inertia.

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by chatmasta on Monday March 11, 2024 @07:50PM (#39674518)

Google sells access to TPUs in its cloud platform, so you'd think they would be more open about sharing development and tooling frameworks for TPUs. It's like Borg (closed source, never used outside Google, made them no profit) vs. Kubernetes (open source, used everywhere, makes them profit).

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by fnord77 on Monday March 11, 2024 @11:29PM (#39675874)

> Google has been doing AI before any other company even thought about it

This not even remotely true. SRI was working on AI in various forms long before google existed

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by TwentyPosts on Tuesday March 12, 2024 @03:53PM (#39684155)

Who or what is SRI?

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by fluidcruft on Tuesday March 12, 2024 @11:25PM (#39687804)

See http://www.sri.com

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by fnord77 on Thursday March 14, 2024 @12:30PM (#39705985)

next to NASA, probably the most innovative organization in human history

https://www.sri.com/timeline-of-innovation/

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by danielcampos93 on Monday March 11, 2024 @06:00PM (#39673723)

I feel like I have been hearing that since V1 TPU. I think Google is the perfect solution because they are teams whose job is to take a model and TPUify it. Elsewhere there is no team, so it's no fun.

[ Parent | on HN ]

Re:Who uses Google TPUs for inference in production? by arthurdelerue on Tuesday March 12, 2024 @10:35AM (#39680052)

I agree with that, and I'm not sure they'll be able to improve the stack dramatically by themselves without the open-source community being more involved.

[ Parent | on HN ]