What is the answer?

Compute. The NVIDIA cards have way more tflops than the Mac SoC. A simple RTX 4090 has 16,384 CUDA cores. The M2's GPU has... 38 cores, apparently? The M2 GPU clocks in at 3.6tflops, compared to ~100tflops for the RTX 4090.

And this is just for a consumer GPU, I haven't even touched on the datacenter-grade stuff.

tl;dr the M2 is an underpowered GPU with a lot of RAM close by, while NVIDIA cards are multiple orders of magnitude more powerful but most of the RAM's a bit farther away.

seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.

There is a video [1] benchmarking last year's M1 Ultra against 3080ti for one ML use case and it was 3x slower. Apple MX does have neural engines which are used but not sure how they compare to CUDA cores.

Either way quite a bit better than 30x slower.

[1] https://www.youtube.com/watch?v=k_rmHRKc0JM

In that video they run the Linux experiments over a windows with a virtual machine. And I didn't see the model, but I bet you I can trains a model in a 4090 one 2x faster than in a old 1050 (because i can chose a model which bottle neck could be the data transfering not the actual computation).

> seriously, measuring graphics cards in gigabytes is like measuring battery life in volts. it's the wrong unit.

Yes, but, if you need 48GB to run inference on a model, and you only have 24GB available, you don't get to enjoy the tflops difference.

If nVidia released somewhat lower-performance GPUs but with more VRAM, we could talk. But they're not stupid :)

You can get an A6000 for cheaper than an M2 Ultra. That said it's still only 48GB.

I'm not sure if it's cheaper when you compare the whole machine price, not just GPU, since you can't buy piecewise with the Mac.

The GPU is $4500. That gives you $2000 to build a machine (which you can build a top of the line consumer PC for cheaper than that, I just did recently w/128GB DDR5 and Zen 4 7950X3D).

Cool if you can find it at that price. On my side of the pond, it's around €6,000.

yes you do. you can use DeepSpeed. it's only like 2x slower.

The cuda cores and m2 cores aren't comparable.

A more appropriate comparison is the fp16 performance.

It seems to be 27 tflops for the 38 core M2, and 330 for the 4090.

The more useful for training fp16 with fp32 accumulate is 165 for the 4090, I don't know about the apple one.

You're not wrong that the RTX 4090 will be much more powerful than an M2 Ultra, but you don't have any idea what you're talking about when you're comparing specs.

You might be able to compare the number of CUDA cores to the ALU count of the Apple GPUs. I don't know what that is for M2 Ultra yet, but for the 64 core M1 Ultra each core had 16 execution units and each of those had 8 ALUs, for a total of 8,192 ALUs. The M1 Ultra's FP32 performance was in the ballpark of 21tflops - assuming a ~30% improvement in the M2 Ultra that takes us to ~27tflops. Google suggests that for the RTX 4090 it's 83tflops.

• The GPU in the base M2 has 10 GPU cores and 3.6 tflops

• The GPU in the M2 Ultra has 76 GPU cores and corresponds to 2x M2 Max, which has 13.6 tflops; so 27.2 tflops [1,2]

• RTX 4090 has 82.58 tflops [3] (overclocked can reach 100 tflops [4])

While more powerful NVidia cards are not "multiple orders of magnitude more". Rather it seems a 4090 is around 3-4 faster than an M2 Ultra GPU.

Keep in mind that the Apple Silicon chips also have low-precission Neural Engine circuits for inference of neural nets. For the M2 Ultra they claim 31.6 tops [1].

[1] https://www.apple.com/newsroom/2023/06/apple-introduces-m2-u...

[2] https://www.notebookcheck.net/Apple-unveils-M2-Pro-and-M2-Ma...

[3] https://www.techpowerup.com/gpu-specs/geforce-rtx-4090.c3889

[4] https://videocardz.com/newz/overclocked-nvidia-rtx-4090-gpu-...

The RTX 4090 has 330.3 TOPS with INT8 precision, so for inference workloads it is still a magnitude faster

Since we are talking about AI, you should use Tensor Core Tops instead of regular flops. That number for 4090 is 330. So more than 10x of M2 Ultra

Max to ultra doesn't scale linearly. Max tech on YouTube covered this extensively

Probably true that performance doesn't scale linearly (and I do remember the Max Tech analysis of M1), but I don't have the tflops number for the M2 ultra.

For the M1 the stated values are:

  M1 Ultra [1]: 21 TFLOPs
  M1 Max [2]: 10.6 TFLOPS

Actual performance obviously depends on the type of work-load etc. E.g. Geekbench Ultra is only 50% faster [3]:

  M1 Ultra Geekbench 6: 150260 
  M1 Max Geekbench 6: 108198

[1] https://videocardz.com/newz/despite-apples-claims-m1-ultra-g... [2] https://wccftech.com/m1-max-teraflops-performance-higher-tha... [3] https://browser.geekbench.com/metal-benchmarks

I am speculating the answer is that "Nvidia just works", where Apple may be more nice & hassle to get working with their preferred frameworks/stacks/tools.

Maybe that, but also the Nvidia chips have *vastly* higher performance (see https://resources.nvidia.com/en-us-grace-cpu/grace-hopper-su...). They claim 4 TB/s memory bandwidth and up to 989 single-precision TFlops in tensor mode (67 TFlops for non-tensor ops).

By contrast, M2 Ultra has 800 GB/s memory bandwidth, 31.6 half-precision TFlops in the Neural Engine, and (extrapolating from https://en.wikipedia.org/wiki/Apple_M2), about 27 single-precision TFlops on the GPU.

So 5x memory bandwidth, more than double generic throughput, and at least 32x peak tensor throughput. Sure, the Mac Studio uses much less power, but depending on the application that usually doesn't make up for the speed difference.

CUDA.

A 15 year head start in a category they essentially defined plus an entire generation of executives, developers, and users doesn’t hurt either.

People complain about the “Nvidia tax” but the hardware is superior (untouchable at datacenter scale) and the “tax” turns into a dividend as soon as your (very expensive) team spends hour after hour (week after week) dealing with issues on other platforms compared to anything based on CUDA often being a Docker pull away with absolutely first class support on any ML framework.

Nvidia gets a lot of shade on HN and elsewhere but if you’ve spent any time in this field you completely understand why they have 80-90% market share of GPGPU. With Willow[0] and the Willow Inference Server[1] I'm often asked by users with no experience in the space why we don't target AMD, Coral TPUs (don't even get me started), etc. It's almost impossible to understand "why CUDA" unless you've fought these battles and spent time with "alternatives".

I’ve been active in the space for roughly half a decade and when I look back to my early days I’m amazed what a beginner like me was able to do because of CUDA. I still routinely am. What you’re able to actually accomplish with a $1000 Nvidia card and a few lines with transformers and/or a Docker container is incredible.

That said I am really looking forward to Apple stepping it up here - I’ve given up on AMD ever getting it together on GPGPU and Intel (with Arc) is even further behind. The space needs some real competition somewhere.

[0] - https://github.com/toverainc/willow

[1] - https://github.com/toverainc/willow-inference-server

I recently sat and thought about it, and I made the bold claim to a friend that the development of CUDA over the last 15 years.. just that activity, as a whole, is one of the largest use of human intelligence to ever be focused on any one thing in history. It's less people than worked on getting to the Moon, but if you measure the project in PhD-person-years it is probably off the charts. And if you include all the engineers and scientists that the program affects in the same way you would have included all the subcontractors for the Apollo program, it is way, way bigger.

I think the only reason CUDA isn't talked about like the monumentally important human milestone in technological development that it is, is that it is a pretty abstract thing that is difficult for laypeople to visualize.

This is an excellent point.

I've been accused of being an Nvidia "fanboy" when I touch on this. I attempt to explain:

"Nvidia made the very hard, very expensive commitment to developing and supporting CUDA 15 years ago when this space was in it's infancy (non-existent). They SUNK incredible resources into this gamble/vision to universally support CUDA on every chip and every platform - for 15 years. They didn't achieve their position through shady dealings or luck, they earned it with a decade and a half of investment, focus, and execution."

Granted, they do somewhat abuse the position they have now (as often noted on HN and elsewhere). At the risk of whataboutism I ask: "Show me a corporation that wouldn't. Do you think if AMD had their market share they'd be nice, cuddly good guys?" Microsoft, Intel, Nvidia, Standard Oil, the Phoebus cartel, etc - it goes on and on. Always has and always will. Of course I'm not saying it's a good thing, it's just a fact of the real world.

So what if other companies would similarly abuse their position? If it doesn't excuse Nvidia's behavior in some way then it hardly seems relevant to point out. We still need more competition in this space, or Nvidia parts will become increasingly bad deals (as we've seen for consumer GPUs).

I'm not saying it's a good thing, I'm simply pointing out it's more-or-less universal standard behavior for people and corporations to abuse positions of power unless checked externally - anti-trust action, checks and balances within government, other regulation, etc. As noted the recurring tendency on HN and elsewhere to paint Nvidia as some kind of unique diabolical actor is very strange.

How are Nvidia consumer GPUs a bad deal? When comparing top of the line cards (I'm not going to bother looking elsewhere in the product lines) for 60% more cost (yes, significant) with an RTX 4090 you get:

- Performance that walks all over the 7900 XTX[0].

- The ability to use it to self-host, experiment, learn, etc a never-ending range of ML applications that (as discussed) "just work" as a docker pull away.

With an RTX 4090 you could have stunning gaming performance one minute and then seconds later be running a local LLM, etc. That is tremendously more value than the 60% price difference. If all I wanted to do was gaming and I was price sensitive I'd save myself the $600 and be happy with an AMD GPU. But, looking at market share[1] (at least 80% across desktop gaming and GPGPU) either the value of AMD GPUs is a little known secret (it isn't) or consumers, with the ability for choice, overwhelmingly see the value in Nvidia GPUs.

[0] - https://techguided.com/7900-xtx-vs-rtx-4090/

[1] - https://wccftech.com/q3-2022-discrete-gpu-market-share-repor...

The sheer volume of work of 'just' creating and maintaining a C++-like compiler, highly performant cuBLAS, cufft, cuSOLVER, cuDNN with very high perf, all by their lonesome in closed source shows how far one would have to go to even eat parts of their lunch.

We (who want a real alternative to NVIDIA) either find a way to pool global resources applicable to this level kind of effort on all new accelerator architectures, or we wait for them to stumble, Intel-like. Intel and AMD not pooling their resources on this is self-defeating.

<< I'm often asked by users with no experience in the space why we don't target AMD, Coral TPUs (don't even get me started), etc. It's almost impossible to understand "why CUDA" unless you've fought these battles and spent time with "alternatives".

Would you be willing to elaborate ( ie. I would love to hear you get started )? I absolutely agree that some competition is needed in this space. I am absolutely not an expert so it is hard for me to understand why there is no real alternative to CUDA. Are they just too hard to set up? Not popular enough to have any support?

Probably the best reference is HN itself. Look at the dozens of GPGPU projects, articles, papers, etc that hit the front page in any given week. Then look to see how many of them support AMD/ROCm. Spoiler alert: virtually zero.

That's compelling enough to justify the position here but you can do further research to explore the challenges with other platforms (like ROCm). Just glance at issue trackers for Pytorch, Tensorflow, and higher level projects that (rarely) support ROCm - you will notice a clear trend. Even though CUDA is more capable and outnumbers AMD use 10:1 the issues reported currently are:

Pytorch

Search for ROCm - 2,557 issues open (~10% market share)

Search for CUDA - 5,430 issues open (at least 80% market share)

Even in the limited cases where it's attempted the capability and experience with AMD/ROCm is significantly worse, to the point of "almost no one even bothers anymore" (see my top paragraph).

A "prosumer" desktop is just plain not up to the job. The Mac Studio is comparable to a high-end gaming PC. It is not really a workstation and definitely isn't a server. Its 16 performance cores and 192 GiB of non-ECC memory cannot compete with nearly 200 cores and multiple terabytes. Its GPU compute and memory bandwidth are many times less than what Nvidia can build at the high end.

The Studio would run large workload many times slower than a high-end server, if it could run them at all.