It's definitely not happening at the high end of the market (NVIDIA A100s with 40GB or 80GB of RAM).

The cards that were used for mining have since crashed in terms of prices, but those were always gamer cards and very rarely Datacenter cards.

The market segmentation is likely a result of Nvidia's monopoly position. They double the RAM and flops, improve the thermals and housing and sell for ten fold the price. It doesn't make sense to me. A cheap 4090 theoretically outperforms even the A6000 RTX Ada. https://timdettmers.com/2023/01/30/which-gpu-for-deep-learni...

Nvidia needs to satisfy gamers, who individually can't spend more than a few $k on a processor. But they also have the server sector on lockdown due to CUDA. Seems they can easily make money in both places. Maybe those H100s aren't such a good deal...

If someone understands these dynamics better I'd be curious to learn!

The main limiter in the data center setting is licensing, interconnects, and ram.

By contract - you can’t sell 4090s in a data center. You’ll find a few shops skirting this, but nobody can get their hands on 100k 4090s without raising legal concerns.

Likewise, nvidia A100s have more than a few optimizations through nvlink which are only available on data center chips.

Lastly, per card memory matters a lot Nvidia has lead the market on the high end here.

Nope, this is about it. They try to force the larger users into the expensive cards by prohibiting datacenter use in the driver EULA. This works sufficiently well in America, but it also means that you can find German companies like Hetzner that will happily rent you lots of consumer cards.

(There are also some density advantages to the SMX form factor and the datacenter cards are passively cooled so you can integrate them into your big fan server or whatnot. But those differences are relatively small and certainly not on their own worth the price difference. It's mostly market segmentation.)

I understood this as $/FLOP, I think it's plausible that that has been happening.