Considering how long it took mid level consumer cards to beat my $600 1080, you're way more optimistic than I am.

The main moat here is VRAM, not raw compute power.

So... Four years? The 1080 launched in 2016, and the 3070 launched in 2020, for $100 cheaper — the launch price of the 1080 was $699, and the 3070 was $599. The 3070 easily trounced the 1080 in benchmarks.

The 3060 effectively matched a 1080 at $329 in 2021 (and has 50% more VRAM at 12GB instead of the 1080's 8GB), so call it five years if the 3070 isn't mid-range enough.

The 3060 Ti launched in 2022 at $399 and handily beat the 1080 on benchmarks, so call it six years if you want the midrange card to beat (not just match) the previous top-of-the-line card, and if a *70 card doesn't count as midrange enough. Less than a decade still seems like a reasonable claim for a midrange card to beat a top-of-the-line card.

are you conveniently forgetting how none of those cards were actually available for consumers to buy

The 1080 was impossible to buy at launch too and was sold out for months. And the 3060 is easy to buy!

The 3060 is easy to buy NOW, since cryptomining has crashed, and the 40x0 GPUs have become available (though mostly still above MSRP).

The 3060 was only readily available quite recently, so it's about 6 years from ready availability of the 1080 at $600 to 3060.

Taking 6 years to double the perf/$ implies that it would take ~42 years for a $40000 H100 to reach mid-range levels. Assuming scaling, particularly VRAM, holds.

And plus it would be getting really close to the Landauer limit by that point.

Given how nvidia has almost no competition, it just seems unlikely that nvidia decides to stop milking the enterprise and they will continue to lock 40GB+ cards behind ludicrous price points

Nvidia may not need to price compete immediately, but if LLMs drive demand for more capable consumer hardware they can either:

(1) make a whole lot of money fulfilling that demand, or

(2) leave an unmet demand that makes it more attractive for someone else to spend the money to field a solution.

(1) seems an attractive choice even if it wasn’t for the potential threat of (2).

>but if LLMs drive demand for more capable consumer hardware they can either:

That's a big if. I imagine people who will buy GPUs to run LLMs locally are either researchers or programmers. I am imagine most consumer focused solutions will be cloud first. That is a much smaller market than gamers and nvidia wouldn't want to cannibalize their datacenter offerings by releasing something cheaper. It's far better for them to sell high tier GPUs to Amazon and let them "rent" it out to researchers and programmers.

It wasn't ML but gaming that drove the demand for GPUs, and ML sort of rode in the slipstream (same for crypto). Later demand for crypto hashing drove GPU sales as much as gaming but that is now over. So unless either (1) ML by itself can present a demand as large as either crypto or gaming or (2) Crypto or gaming can provide a similar demand as they've done in the past the economies of scale that drove this will likely not be reached again. If they do however the cost of compute will come down drastically and that in turn may well drive another round of advances in models for the larger players.