News for hackers. Stuff that matters.
Re:Ironwood: The first Google TPU for the age of inference by miki123211 on Wednesday April 09, 2025 @03:35PM (#43636728)
Wow, you guys needed a custom ASIC for inference before CNNs were even invented?

What were the use cases like back then?

Re:Ironwood: The first Google TPU for the age of inference by huijzer on Wednesday April 09, 2025 @03:51PM (#43636964)
According to a Google blog post from 2016 [1], use-cases were RankBrain to improve the relevancy of search results and Street View. Also they used it for AlphaGo. And from what I remember from my MSc thesis, they also probably were starting to use it for Translate. I can't find any TPU reference in the Attention is All You Need or BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, but I have been fine-tuning BERT in a TPU at the time in okt 2018 [2]. If I remember correctly, the BERT example repository showed how to fit a model with a TPU inside a Colab. So I would guess that the natural language research was mostly not on TPU's around 2016-2018, but then moved over to TPU in production. I could be wrong though and dgacmu probably knows more.

[1]: https://cloud.google.com/blog/products/ai-machine-learning/g...

[2]: https://github.com/rikhuijzer/improv/blob/master/runs/2018-1...

Re:Ironwood: The first Google TPU for the age of inference by mmx1 on Wednesday April 09, 2025 @04:48PM (#43637786)
Yes, IIRC (please correct me if I'm wrong), translate did utilize Seastar (TPU v1) which was integer only, so not easily useful for training.
Re:Ironwood: The first Google TPU for the age of inference by refulgentis on Wednesday April 09, 2025 @03:48PM (#43636930)
https://research.google/blog/the-google-brain-team-looking-b... is a good overview

I wasn't on Brain, but got obsessed with Kerminology of ML internally at Google because I wanted to know why leadership was so gung ho for it back then: the general sense in the early days was these things can learn anything, and they'll replace fundamental units of computing. I feel this sense is best exhibited externally by ex. https://research.google/pubs/the-case-for-learned-index-stru...

It was also a different Google, the "3 different teams working on 3 different chips' thing reminds me of lore re: how many teams were working on Android wearables before it was formalized.

FWIW it's a very, very, different company now. Back then it was more entrepreneurial, a sort of better version of Wave-era where things would launch randomly. An MBA would find this top-down company in 2025 even more efficient, I find it less - it's perfectly tuned to do what Apple or OpenAI did 6-12 months ago, but not to lead. (disclaimer: worked on Android)

Re:Ironwood: The first Google TPU for the age of inference by marsten on Thursday April 10, 2025 @04:31AM (#43641930)
Google was changed by two things, neither of which were much fun. But very understandable.

One was the transition to a mature product line. In the early days it was about how do we do cool new things that will delight users: Gmail, Google Maps (Where 2), YouTube. The focus was on user growth and adoption.

Then growth saturated and the focus turned to profitability: Getting more value out of existing users and defending the business. That shift causes you to think very differently, and it's not as fun.

The second was changing market conditions. The web grew up, tech grew up, and the investment needed to make a competitive product skyrocketed. Google needed more wood behind fewer arrows and that meant reining in all the small teams running around doing kooky things. Again not fun, but understandable.

Re:Ironwood: The first Google TPU for the age of inference by dekhn on Wednesday April 09, 2025 @05:44PM (#43638362)
As an aside, Google used CPU-based machine learning (using enormous numbers of CPUs) for a long time before custom ASICS or tensorflow even existed.

The big ones were SmartASS (ads serving) and Sibyl (everything else serving). There was an internal debate over the value of GPUs with a prominent engineer writing an influential doc that caused Google continue with fat CPU nodes when it was clear that accelerators were a good alternative. This was around the time ImageNet blew up, and some eng were stuffing multiple GPUs in their dev boxes to demonstrate training speeds on tasks like voice recognition.

Sibyl was a heavy user of embeddings before there was any real custom ASIC support for that and there was an add-on for TPUs called barnacore to give limited embedding support (embeddings are very useful for maximizing profit through ranking).

Re:Ironwood: The first Google TPU for the age of inference by mmx1 on Wednesday April 09, 2025 @09:11PM (#43639668)
What I've heard is that the extrapolation of compute needed so many additional CPU servers to keep running the existing workload types that it obviously justified dedicated hardware. Same for video encoding accelerators[1].

[1]: https://research.google/pubs/warehouse-scale-video-accelerat...

Re:Ironwood: The first Google TPU for the age of inference by dekhn on Thursday April 10, 2025 @12:13PM (#43645380)
yes, that was one of the externally shared narratives. The other part is that google didn't want to be beholden to nvidia GPUs, since they have an associated profit margin that is higher than TPUs, as well as resource constraints (total amount of GPUs shipping at any given time).

Another part that was left out was that Google did not make truly high speed (low-latency) networking and so many of their CPU jobs had to be engineered around slow networks to maintain high utilization and training speed. Google basically ended up internally relearning the lessons that HPC and supercomputing communities had already established over decades.

Re:Ironwood: The first Google TPU for the age of inference by miki123211 on Thursday April 10, 2025 @01:20PM (#43646089)
> The big ones were SmartASS (ads serving) and Sibyl (everything else serving).

Ah, the days when you, as a tech company employee, could call a service "SmartASS" and get away with it...