I feel like I'd have a very hard time designing a $7000 system which doesn't completely blow the Mac Pro out of the water in terms of CPU and GPU performance. Do you have some examples of $7000 systems which are beaten by the $7000 Mac Pro's cut-down M2 Ultra in terms of CPU or GPU power?

RTX 6000 Ada alone is $7000. Don’t see how you’re “completely blowing the Mac Pro out of the water” at that price point.

The 4090, which seems like it's blowing the M2 Ultra out of the water, is $1600 though.

Great for Cyberpunk 2077, but not great if you need more than 24 GiB.

Considering the RTX 6000 ADA is like $8k, you could add three 4090s for half the cost.

You know what, that's actually fair. If your workload needs tons of GPU memory, nothing probably beats the Mac Pro's up to 192GB.

Except, I guess, for the Mac Studio.

Is there a workload that actually needs more than 24GB of VRAM that can't be offloaded to regular RAM.

I honestly don't have any examples out of my head. LLM inference is compute bound AFAIK.

LLM inference is GPU bound and VRAM bound. Given quantization however, 2x3090 or 4090 (48GB) is enough VRAM to load 65B quantized llama derivatives (30-40GB). With exllama you can get maybe 20-30 tok/s with dual 4090s on ~1200W. With an M2 Ultra maybe 10 tok/s on ~178W.

(I don't have these devices but have been researching LLM inference performance in the interest of buying a machine for them.)

Advantage goes to M2 Ultra if you ever might need more than 48GB VRAM. I think it's unlikely nvidia is going to release a significantly higher VRAM consumer card anytime soon, since that's what their A100s are for.

If only the big ML stuff didn't all use CUDA. Guessing that world won't transition to Metal compute any time soon.

Are there any such non AI use cases? For AI anyway its all Nvidia.

You'll still be limited by memory. If Apple were to create a Jade 4c chipset, (two Ultra SoC), you'd only have 384GB of RAM.

Depends on what you do. Let's say Apple makes an M3 "Extreme" and sets max memory to 512GB of RAM.

That's 512GB of high bandwidth RAM that you'd have to piece together 7 Nvidia H100 80GB cards to match. Each H100 80GB card costs $30,000+. Seven of them would cost $210k alone.

Now of course 7 H100s would have more compute power, but if what you need is a lot of high bandwidth RAM such as a large LLM or highly detailed scene rendering, then maybe an M3 "Extreme" would have the best value out there, by far.