I agree, GPUs can still generate more token per second per dollar, but what is new and great about the high-end M1 and M2 is Apple offering this much memory bandwidth on a general purpose CPU, thus immediately available to all software running on the CPU.

Unless you calculate power usage I’d bet

I would wager the M2 is more energy efficient than a 4090.

It fully depends on the workload. The 4090 itself draws around 450w at load, and the M2 Ultra peaks around 300w. If your workload is >1.5x faster on Nvidia hardware, then it's per-prompt efficiency probably beats out the M2 Ultra.

Not to mention that you can trivially set a 300w power cap on the 4090 and still get 75-80% of peak FLOPS.

All valid points, I think I looked at it too trivially and fired an opinion from the hip.

Looking at the prices, The 4090 is about 3/4 of the price of a base model M2 Ultra Mac Studio, which has 64GB RAM. With the rest of the PC to go with the graphics card, it's about 7/8s of the price of an Ultra. Then you have the compatibility. Do you want your software to be CUDA compatible or Metal compatible? If you're writing the software, maybe you want both!

The budget option is to go with a used 3090, which still has greater memory bandwidth than the M2 Ultra.

In terms of FP16 Flops on the GPU, you have:

M2 Ultra, 27 TFLOPS

RTX 3090, 35 TFLOPS

RTX 4090, 82 TFLOPS

https://www.cpu-monkey.com/en/igpu-apple_m2_ultra_76_core

https://www.techpowerup.com/gpu-specs/geforce-rtx-3090.c3622

https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889