I doubt it will change much, inference (not training) is very profitable and demand for inference is quite high. See: Mistral serving GLM on their own servers.

Pure inference would have to compete with other service providers and even local hardware that becomes capable of running models for a price if hardware demand by openAI and Anthropic dwindles.

That is a very commodify market and as such I expect power cost per query ends up setting the pricing.

> if hardware demand by openAI and Anthropic dwindles

Would it ever? Demand for inference (and I suppose the same applies for training) _increases_ when models get cheaper.

I suppose real risk is that one player aggressively cuts margins (say, AWS) and others follow suite. However the release of increasingly more capable, exclusive closed-weight models prevents this to an extent.

They are probably too big to fail already (and too friendly with the current US gov). And even if they fail, open-weight models will take over as the usefulness of the tech behind has been proven.

In that unlikely case, open-weight hosting providers (and Google and Meta and SpaceXAI) would pick up the slack.

The only chance of memory demand going down would be breakthroughs in model size reduction.

The catchup models are all basically distillations of the sota models. Without the next training cycle things are going to stagnate. And as all those companies you mentioned are all linked to OpenAI and Anthropic and relying on hosting deals and things they are all going to be in the ringer when things to go south.

So on the one hand you have all the sota model makers doing investor expectation management in saying wet need a slowdown for safety (may or may not be true, but also means they don’t spend on the next training cycle before IPO? Could be making their books look better too?) and on the other we have a sense that the models aren’t yet at the stopping place where we can just not train another cycle and use distillations of the current generation?

> The catchup models are all basically distillations of the sota models

Source/citation?

https://www.dwarkesh.com/p/john-beren-charlie Is one of many and it describes how the dynamics well and talks about how proxy logs help.

Thoughts/opinions aren't source.

The people in that interview stating this are working at the forefront of open source models and research and includes a cofounder of OpenAI. They don’t say it like it is conjecture.

Also gpu speedup. If vera Rubin is 7x faster then to serve same number of tokens you need 1/7 the memory. Did I get that right?

Sure, and they'll be cognizant of that risk when considering expansion. The lessons of the dotcom fiber boom aren't that distant in memory.

If AI companies have allocation of memory, can't they just sell the memory allocation if they are tight on cash?

I expect them to sell everything if they go bust to recoup what little they can for investors. That would double whack hardware manufacturers are their final consumers are now both leaving the markets as buyers and entering as competitors.

Well, but to who? And for pennies on the dollar, most likely. Also not all ram is made equal. HBM for AI cards isn't LPDDR5 in iPhones.

HBM consumer GPUs.

RAM production is easier to transfer same with flash

But sell it to consumers, where there is pent up demand

Given that even small GPU repair companies in China can put GPU chips on new boards with double the original memory, I would not be surprised to see the same people selling HBM from scrapped AI data centers acting as RAM on adapter boards. ;-)

I don’t think they will. There is too much money in it now.

Only last year Microsoft admitted to having GPUs in inventory that they couldn't power, due to a lack of electricity. So much of this production capacity could very well go into making memory chips for GPUs sitting in a warehouse.

The amount of money spend on the AI hype train is crazy. Meanwhile we're wasting fab time making chips that might never be powered on. It's absolutely insane that there are more money to be made on hardware for AI that may never be used, rather than producing a product that consumers and businesses need right now.

But wait, I thought we all agreed that capitalism is the most efficient way to distribute goods in an economy!

By goods you mean wealth vertically to the oligarchs, then yes.

I'd argue that this is a form of financial engineering that is somewhat removed from "true" capitalism.

Don't worry, the Trump administration is addressing this by lifting pollution limits for data centers.

They're too big to fail at this point. US taxpayers will bail them out if push comes to shove.

I certainly won't be. The US Government that allocates my tax dollars might though.