I was thinking about that (I definitely agree with you on the software and data angle).

But when you think about it it's actually a bit more complex. Right now (eg) OpenAI buys GPUs from (eg) NVidia, who buys HBM from Samsung and fabs the card on TSMC.

Google instead designs the chip, with I assume a significant amount of assistance of Broadcom - at least in terms of manufacturing, who then buys the HBM from the same supplier(s) and fabs the card with TSMC.

So I'm not entirely sure if the margin savings are that huge. I assume Broadcom charges a fair bit to manage the manufacturing process on behalf of Google. Almost certainly a lot less than NVidia would charge in terms of gross profit margins, but Google also has to pay for a lot of engineers to do the work that would be done in NVidia.

No doubt it is a saving overall - otherwise they wouldn't do it. But I wonder how dramatic it is.

Obviously Google has significant upside in the ability to customise their chips exactly how they want them, but NVidia (and to a lesser extent) AMD probably can source more customer workflows/issues from their broader set of clients.

I think "Google makes its own TPUs" makes a lot of people think that the entire operation in house, but in reality they're just doing more design work than the other players. There's still a lot of margin "leaking" through Broadcom, memory suppliers and TSMC so I wonder how dramatic it is really is

My take is it's the inference efficiency. It's one thing to have a huge GPU cluster for training, but come inference time you don't need nearly so much. Having the TPU (and models purpose built for TPU) allows for best cost in serving at hyperscale.

Yes potentially - but the OG TPUs were actually very poorly suited for LLM usage - designed for far smaller models with more parallelism in execution.

They've obviously adapted the design but it's a risk optimising in hardware like that - if there is another model architecture jump the risk of having a narrow specialised set of hardware means you can't generalise enough.

Prefill has a lot of parallelism, and so does decode with a larger context (very common with agentic tasks). People like to say "old inference chips are no good for LLM use" but that's not really true.

NVidia is operating with what, 70% gross margin? That’s what Google saves. Plus, Broadcom may be in for the design but I’m not sure they’re involved in the manufacturing of TPUs.

Broadcom does the physical design and sources a huge amount of the IP like serdes blocks. TSMC manufactures the chips.

What a wild situation to have a significant part of Earth’s major economies be directly reliant, not on one country, but on one building in the world.

Yeah this is a bummer. If it goes south everyone in power will also have perfect hindsight and say they saw it coming because obviously you shouldn't have this much built on such a small footprint. And yet...

> Yeah this is a bummer. If it goes south everyone in power will also have perfect hindsight and say they saw it coming because obviously you shouldn't have this much built on such a small footprint. And yet...

It'll be true, everyone does see it coming (just like with rare earth minerals). But the market-infected Western society doesn't have the maturity to do anything about it. Businesses won't because they're expected to optimize for short-term financial returns, government won't because it's hobbled because biases against it (e.g. any failure becomes a political embarrassment, and there's a lot of pressure to stay out of areas where businesses operate and not interfere with businesses).

America needs a lot more strategic government control of the economy, to kick businesses out of their short-term shareholder-focused thinking. If it can't manage that, it will decline into irrelevance.

When USSR fell there was a lot of talk about how it was meant to be since the US system is the best of all the terrible systems. It deserved to win and the USSR system deserved to die.

>America needs a lot more strategic government control of the economy, to kick businesses out of their short-term shareholder-focused thinking. If it can't manage that, it will decline into irrelevance.

If it is meant to be then its meant to be. If the US decides to cling to its old system and it fails well then we would know that it wasn't the best system after all. Humankind will keep moving forward even if it means that another continent controls the show.

Just by removing nvidia's profits Google gets their TPUs for like 25-30% of what they cost to get them from nvidia, assuming similar cost structures. Google's cost structures are probably higher than nvidia's so realistically theyre probably paying around 50% of nvidia charges, but thats still billions of dollars a year and allows them to create a product that is tailor made for their needs.

Google has the time, money, TPUs, and ability to siphon talent. It'll be an unsexy slog to the top, but they'll get there eventually.

While I think Gemini is the worst of the three big competitors, Waymo is an superb example of this talent. Kudos to Google engineers for producing so many diamonds despite producing many terrible flops over the years. We might find out their system of organization was the best one after all.