GPU scarcity is a really poor proxy for the health of the AI market. I don’t know if I should be surprised that someone so bullish about it doesn’t understand this.

you should not be surprised. paper millions already borrowed against at stake

Did you read the OP? Because Nvidia gave money to people who aren't you, so those people could give the money back to Nvidia for GPUs. In other words they gave away all of their GPUs in exchange for stakes in startups, because (1) they could not sell that quantity of GPUs at market prices ordinarily and (2) they were betting that VC hype and the perception of high demand generated from doing so would be worth more than selling GPUs normally.

But presumably those gpus are being actually used by those companies? Even if their prices are artificially inflated? Or are these buyers sitting on piles of unused gpus to make demand look good?

I mean even Satya Nadella famously said that Microsoft has plenty of GPUs just nowhere to put them. There are definitely plenty of unused GPUs out there. Jensen is also a big fan of scarcity (said in multiple interviews) so obviously he’d want to keep it up at all costs.

> These GPUs are probably in use now,

A different question might be, will these companies grow over the new few years?

If I give you 10 Million GPU in exchange for 25% of the company and the company goes bust or has negative growth (and thus is forced to sell assets), effectively I just put 10M GPUs on the market with a huge discount.

At a grand enough scale, this undermines the GPU marker and one could argue that Nvidia has been doing this on a massive scale.

> Or are these buyers sitting on piles of unused gpus?

Yes, they are literally doing exactly that. Noting that they aren't truly "buyers" so much as "gift recipients" (in exchange for ownership stake, but the value of their remaining ownership also goes up if these gifts generate sufficient hype for investors to invest more). GPU production is outstripping datacenter capacity.

> But presumably those gpus are being actually used by those companies?

But, even if they were all being used, this wouldn't mean much. The central point is: suppose Nvidia has produced 10 million[1] datacenter-class GPUs during the LLM bubble. Nobody can afford to, or is willing to, buy 10 million datacenter-class GPUs because the economics don't work out. But if they gift 10 million GPUs, and frame it like they sold the GPUs (a framing that is effective -- although this has been an open secret, this is being reported like news by the WSJ, and to the majority of their readers, it is news), it makes Nvidia GPUs appear to be extremely desirable. Even though, logically, an item that you have to give away because it's not economically worth its market price is the exact opposite. The fact that you can use goods that you've effectiely been given for free does not increase the actual economic value of those goods.

[1] (hypothetical number, it's vaguely around this order of magnitude)

Why would they sit on piles of unused GPUs when they could rent them out or use them? Even if they were free that is still not what a business would do with them.

"Why don't they just use their unused GPUs"? - because they literally cannot. GPUs require electricity. They have more GPUs than they can supply electricity to.

https://www.datacenterdynamics.com/en/news/microsoft-has-ai-...

> Microsoft CEO Satya Nadella has said the company has AI GPUs sitting idle because it doesn’t have enough power to install them

https://www.businessinsider.com/amazon-nvidia-ai-gpu-data-ce...

> Some AWS sites had room for more racks but not enough available power to turn them on

> At least one AWS team did not have "a clear forecasted path" for getting the incoming racks into data centers at the time

Do you truly believe that the cost that can be charged for inference is below the cost of electricity?

No. I know for a fact that the take you've stated is incorrect, because I work for a startup that makes 8-digit profit annually on inference, with zero outside investment. I don't know why you've latched on to my comment to put this take into my mouth, because I never said anything resembling it. It can be true that a technology has some use while also being true that there is a bubble around people believing things about the technology that aren't true. The internet was one such example: the dotcom bust did not indicate that internet services were not financially viable, but that there were highly irrational allocations of investment that resulted in a lot of people losing a lot of money.

There is a fundamental difference from being able to profit on running some number of GPUs, and being able to profit from running 1 million GPUs. We do not have a business case for 1 million GPUs. Our startup's premise is that there are economically viable niches to use LLMs as tools, which is in stark contrast to OpenAI and Anthropic who are at least publicly selling the idea that LLMs are a form of genuine intelligence that will replace all human labor if you just give them another $10 billion dollars, and then another $10 billion dollars, and then another $10 billion dollars, .....

You may not have a business case for 1 million GPUs but you are not the only business. You clearly have a decent business case for some level of inference as do many others.

The gap between how many GPUs people have a profitable business case for and how many are actually out there is, however, a problem. Per their revealed financials for the pending IPO, Anthropic is deeply, deeply in the hole relative to their spending commitments, which are nearing $1 trillion over the next 5 years. OpenAI's do not appear to be much better from the more limited information we have about them.

The bubble popping will actually make business cases for GPUs much more common to come by, when businesses can acquire them for closer to their cost of production rather than at the insane markups Nvidia currently lists them for. But that won't do Nvidia much good, because Nvidia's valuation is predicated on being able to "sell" as many GPUs as they can produce at a 10x markup over the cost of production - even if much of what is being "sold" is really being given away for equity in businesses that are not financially viable.

> The gap between how many GPUs people have a profitable business case for and how many are actually out there is, however, a problem.

Is this a hypothetical gap or do you have actual numbers to back this up?

The rest of that paragraph continued on to mention two of the biggest users not actually having a profitable business case for the amount of GPUs they have.

That is a huge logical leap you are making then and it isn't supported by the evidence you gave. Are you suggesting that that two companies having more GPUs than they can use is proof GPUs aren't useful at their current price point?

There is a limit to how many GPUs can be profitable at this price point. Suppose you're running a SaaS. You might be able to charge $20/mo and get a sufficient number of customers to cover X amount of GPUs. This does not mean the market can support X * 100 GPUs profitably. THere is a limit to what the market can bear. THere are not infinite customers, in any niche, who will pay $20/mo for an LLM-based SaaS. At some point you will saturate your market of potential customers and purchasing more GPUs will not benefit you. And this even applies cross-SaaS. There is a finite pool of any amount of customers who will pay $20/mo for LLM services. If someone is subscribing to a competitor, they are less likely to pay $20/mo to you as well. All of this to say, it is certainly possible to oversupply GPUs that have a 10x markup. If the price is closer to production cost, then maybe they're cheap enough that you can offer a $10/mo plan profitably, and reach a much wider pool of customers and benefit from more GPUs. But there is a finite number of GPUs at which point the market for a sustainable business cases is satiated.

(Consumer sub prices were used for illustrative purposes but this obviously applies to token and enterprise spending as well. Although they're being a little loose due to FOMO, enterprises will not sign blank checks forever if the promised "replace all human labor" does not materialise)

Also, it goes without saying that a GPU that can not even be powered on for lack of power capacity is not being used profitably, at any price point. It could not be more apparent there is an oversupply relative to its current market price, which is a direct result of Nvidia giving away boatloads of GPUs for stakes in unprofitable companies.

OpenAI is in much worse shape because at least Anthropic has the growth to try to justify the spend. I can't see how the economics work but it is at least in some kind of margin of error.

OpenAI just seems like a black hole for money.

I think what you are describing is the sociological/ecological function of a bubble. We pull an unreasonable amount of demand forward, someone else pays for it and when the bubble implodes the good companies that survive the stress test get a free lunch.

Best of luck at the start up. That all sounds right to me. Real intelligence is specialized and ecological. The grand irony to me is that I would love to be an organism that can walk the way I do but also be able to fly like a bird and swim like a shark.

The problem is that organism would be massively too expensive to survive.

OpenAI's are worse and the market thinks so; they are having trouble raising the money they want, the way they want.

Agree with the rest of this though. This whole idea that Nvidia is likely to get much of a normal return from their vendor financing strikes me as tenuous, and what they really are getting is the creation of the perception of scarcity.

One day soon, the notion that GPU scarcity is a proxy for AI demand will finally lose its currency and I think that will be the start of the reckoning.

It’s not the digital world where you just duplicate an object in memory. You need to find the power somewhere (new contract, generator installation), route it, add management tools (and maybe technicians), and then plug the new gpus. It’s all physical work, which also does cost money, take time, and maybe constrained.

ah this is the category error of the whole bubble.

We are so use to the digital world and the efficiency of electronic markets that it is hard to think of physical constraints in the real world.

> Do you truly believe that the cost that can be charged for inference is below the cost of electricity?

That isn't the issue - it's that building brand new datacentres + the power generation to run them takes a massive amount of time/permitting/money, and by the time those datacentres actually come online, you can be sure that the price pressure on inference will be sharply negative.

Nvidia does lease them back from some of these customers. But it's not clear even Nvidia is plugging them all in.

Car companies do that all the time ever since banking licenses have been issued to non-financial institutions.

Nothing new, nobody pays a car cash anymore

I wonder if the economics of the two industries is similar enough to be useful here. Car demand itself seems fairly stable and repossession is a way of recovering value in cases of default. Credit scores of individuals purchasing cars being tied to salary/wages vs. startups without much history. I just wonder if the analogy isn't fraught with issues.

Most of the GPUs that have been sold haven't been plugged in because there isn't enough electricity in the world

replace world with US

there is plenty of electricity in china

There's plenty. Electricity (and energy) is absurdly cheap in the modern world. 1GJ is roughly 10$. An H100 running 24/7 takes 20GJ/year, so that's 200$, which is orders of magnitude less than its retail price.

If energy wasn't cheap, you wouldn't see people driving large vanity SUVs around in the least economic way possible.

Something something something pride before a fall something.

It’s very much not true in all geographies.

It also requires major infrastructure to deliver that energy, which takes money and time to build, as do the datacenters.

"If cake weren't cheap, me and the other aristocrats wouldn't eat it daily"

Around 0.05-0.15% of the world population own such large SUVs.

Just FYI, there are around 360 mil SUV's on the road today around the world. Let's say 1/3 are "large", 120 mil. So, out of population of 8.3 bil, that's 1.4% or apx 3% of households. :-)

You mean there are places and times when electricity is cheap and available, and it if would last 24/365, then it would take 200$ to run such a card. The thing is: it is not available 24/356 in that quantity at that price.

Can I get a source on that please? I think the talk of not having enough power for the AI rollout refers to future needs not current demand.

There's plenty of electricity to run every GPU in the world, just maybe not enough electricity to run every GPU in the world and have poor people afford AC.

Great! How steep a discount can I get to take them off the hands of people who can't plug them in?

Right? This fever dream that there are boxes upon boxes of unused cards (some even claiming on this thread that it is 50% of GPUs sold) is utterly delusional. Some people want AI to go away so badly that they temporarily lose the ability to reason.

You're giving them a lot of leniency. Personally I get the impression many never had the ability to think in the first place, just imitate it by regurgitating what they've heard from others.

I try and give a lot of leniency because generally these people aren't stupid or meaning to be unpleasant, they are just scared. Which means sympathy, kindness and understanding is the likely cure. You can't reason someone out of a position they didn't reason themselves into.

Because most hyperscaler projects are stalling. The number of datacenter chips number was way higher in Zitron's report.

While your datacenter is not built, guess where the AI chips are? That's correct. In boxes.

> While your datacenter is not built, guess where the AI chips are? That's correct. In boxes.

Are you sure about that? Might just as well be in Excel sheets only...

I'm willing to believe Amazon is sitting on bunch of unused GPU[1].

[1]https://www.techspot.com/news/114315-dark-gpus-amazon-show-l...

With the recent craziness, perhaps not all of it is as far from the truth as we might think? There could be quite a gap between units "sold" and units actually made...

If it were to be worse than channel stuffing — that is to say, if Nvidia has sold GPUs that exist only on paper — that would be beyond bad.

FWIW we absolutely know for sure that both AWS and Microsoft have racks upon racks of GPUs that are unused, if the cards are installed at all.

(AWS could have $20B of unused GPUs at this point — it's widely reported that there was at least $5B in August, with up to $3B more arriving each month)

https://www.businessinsider.com/amazon-nvidia-ai-gpu-data-ce...

We know that Satya Nadella has been open about how one of his big problems (as a CEO) is that he has bought GPUs he cannot plug in anywhere.

So 50%? Who knows. But if AWS and Microsoft can't plug them all in, why assume anyone else can?

The clear indication is that cold/dark GPUs delivered to major customers, GPUs that Nvidia has sold but still has on its shelves (buy and hold), and GPUs that Nvidia has sold, leased back from customers and may or may not have plugged into their own cloud, together represent a large enough proportion of sales to have pushed up the prices of GPUs and caused scarcity that is somewhat artificial.

If you take the scarcity of GPUs to be a solid proxy for assessing compute demand — or even for the sales of GPUs to be a solid proxy — then you are making a dangerous bet, IMO.

Step 1: wait for the financial bubble to pop

Step 2: still have money even though the financial bubble popped

Step 3: just buy them normally, at market prices

This is a preposterous statement to make.