News for hackers. Stuff that matters.
Re:Apple M3 Ultra by TheTxT on Wednesday March 05, 2025 @09:22AM (#43266769)
512GB unified memory is absolutely wild for AI stuff! Compared to how many NVIDIA GPUs you would need, the pricing looks almost reasonable.
Re:Apple M3 Ultra by InTheArena on Wednesday March 05, 2025 @09:33AM (#43266937)
A server with 512GB of high-bandwidth GPU addressable RAM in a server is probably a six figure expenditure. If memory is your constrain, this is absolutely the server for you.

(sorry, should have specified that the NPU and GPU cores need to access that ram and have reasonable performance). I specified it above, but people didn't read that :-)

Re:Apple M3 Ultra by Numerlor on Wednesday March 05, 2025 @09:36AM (#43266975)
A basic brand new server can easily do 512gb. Not as fast as soldered memory but it should be maybe mid to high 5 figures
Re:Apple M3 Ultra by la_oveja on Wednesday March 05, 2025 @09:44AM (#43267103)
Re:Apple M3 Ultra by InTheArena on Wednesday March 05, 2025 @09:48AM (#43267188)
That's CPU only memory, not high bandwidth, and not addressable by the GPU.
Re:Apple M3 Ultra by KeplerBoy on Wednesday March 05, 2025 @10:44AM (#43268092)
addressable is a weird choice of words here.

CUDA has had managed memory for a long time now. You absolutely can address the entire host memory from your GPU. It will fetch it, if it's needed. Not fast, but addressable.

Re:Apple M3 Ultra by p_ing on Wednesday March 05, 2025 @11:32AM (#43268810)
Windows has been doing this since what... the AGP era? Though this is a function of the ISA rather than the OS.
Re:Apple M3 Ultra by jeffbee on Wednesday March 05, 2025 @09:55AM (#43267313)
There isn't anything particularly high-bandwidth about Apple's DDR5 implementation, either. They just have a lot of channels, which is why I compared it to a 24-channel EPYC system. I agree that their integrated GPU architecture hits a unique design point that you don't get from nvidia, who prefer to ship smaller amounts of very different kinds of memory. Apple's architecture may be more suited to some workloads but it hasn't exactly grabbed the machine learning market.
Re:Apple M3 Ultra by buildbot on Wednesday March 05, 2025 @11:13AM (#43268529)
M3 Ultra has 819GB/s, and a single epyc cpu with 12 channels has 460GB/s. As far as I know, llama.cpp and friends don’t scale across multiple sockets so you can’t use a dual socket Turin system to match the M3 Ultra.

Also, 32GB DDR5 RDIMMS are ~200, so that’s 5K for 24 right there. Then you need 2x CPUs, at ~1K for the cheapest, and you need 2, and then a motherboard that’s another 1K. So for 8K (more, given you need a case, power supply, and cooling!), you get a system with about half the memory bandwidth, much higher power consumption, and very large.

Re:Apple M3 Ultra by adrian_b on Wednesday March 05, 2025 @12:43PM (#43269762)
Partial correction, an Epyc CPU with 12 channels has 576 GB/s, i.e. DDR5-6000 x 768 bits. That is 70% of the Apple memory bandwidth, but with possibly much more memory (768 GB in your example).

You do not need 2 CPUs. If however you use 2 CPUs, then the memory bandwidth doubles, to 1152 GB/s, exceeding Apple by 40% in memory bandwidth. The cost of the memory would be about the same, by using 16 GB modules, but the MB would be more expensive and the second CPU would add to the price.

Re:Apple M3 Ultra by buildbot on Wednesday March 05, 2025 @01:39PM (#43270475)
Ah, I didn’t realize they’d upped the memory bandwidth to DDR5-6000 (vs 4800), thanks for the correction!

The memory bandwidth does not double, I believe. See this random issue for a graph that has single/dual socket measurements, there is essentially no difference: https://github.com/abetlen/llama-cpp-python/issues/1098

Perhaps this is incorrect now, but I also know with 2x 4090s you don’t get higher tokens per second than 1x 4090 with llama.cpp, just more memory capacity.

(All if this only applies to llama.cpp, I have no experience with other software and how memory bandwidth may scale across sockets)

Re:Apple M3 Ultra by adrian_b on Wednesday March 05, 2025 @02:33PM (#43271167)
The memory bandwidth does double, but in order to exploit it the program must be written and executed with care in the memory placement, taking into account NUMA, so that the cores should access mostly memory attached to the closest memory controller and not memory attached to the other socket.

With a badly organized program, the performance can be limited not by the memory bandwidth, which is always exactly double for a dual-socket system, but by the transfers on the inter-socket links.

Moreover, your link is about older Intel Xeon Sapphire Rapids CPUs, with inferior memory interfaces and with more quirks in memory optimization.

Re:Apple M3 Ultra by buildbot on Wednesday March 05, 2025 @02:58PM (#43271512)
Yes, I believe in theory a correctly written program could scale across sockets, depending on the problem at hand.

But where is your data? For llama.cpp? For whatever dual socket CPU system you want. That’s all I am claiming.

Re:Apple M3 Ultra by adrian_b on Wednesday March 05, 2025 @03:33PM (#43271937)
Googling for what you ask has found immediately this discussion:

https://github.com/ggml-org/llama.cpp/discussions/11733

about the scaling of llama.cpp and DeepSeek on some dual-socket AMD systems.

While it was rather tricky, after many experiments they have obtained an almost double speed on two sockets, especially on AMD Turin.

However, if you look at the actual benchmark data, that must be much lower than what is really possible, because their test AMD Turin system (named there P1) had only two thirds of the memory channels populated, i.e. performance limited by memory bandwidth could be increased by 50%, and they had 16-core CPUs, so performance limited by computation could be increased around 10 times.

Re:Apple M3 Ultra by buildbot on Wednesday March 05, 2025 @06:57PM (#43274457)
Cool, I didn’t find that one! Thanks.

A single 192 core Epyc is 11k by itself, so I’d probably go for the simpler integrated M3 ultra solution…

Re:Apple M3 Ultra by aurareturn on Wednesday March 05, 2025 @01:39PM (#43270477)
CPUs do not have enough compute typically. You'll be compute bottlenecked before bandwidth if the model is large enough.

Time to first token, context length, and tokens/s are significantly inferior on CPUs when dealing with larger models even if the bandwidth is the same.

Re:Apple M3 Ultra by adrian_b on Wednesday March 05, 2025 @02:45PM (#43271332)
One big server CPUs can have a computational capability similar to a mid-range desktop NVIDIA GPU.

When used for ML/AI applications, a consumer GPU has much better performance per dollar.

Nevertheless, when it is desired to use much more memory than in a desktop GPU, a dual-socket server can have higher memory bandwidth than most desktop GPUs, i.e. more than an RTX 4090, and a computational capability that for FP32 could exceed an RTX 4080, but it would be slower for low-precision data where the NVIDIA tensor cores can be used.

Re:Apple M3 Ultra by kiratp on Wednesday March 05, 2025 @07:18PM (#43274577)
Nobody is using FP32 for AI.

INT8, INT4, FP8 and soon FP4

Re:Apple M3 Ultra by adrian_b on Thursday March 06, 2025 @03:22AM (#43277777)
True, but I have compared the FP32 used in graphics computations because for that the throughput information is easily available.

Both CPUs (with the BF16 instructions and with the VNNI instructions for INT8 inference) and the GPUs have a higher throughput for lower precision data types than for FP32, but the exact acceleration factors are hard to find.

The Intel server CPUs have the advantage vs. AMD that they also have the AMX matrix instructions, which are intended to compete for inference applications with the NVIDIA tensor cores, but the Intel CPUs are much more expensive for a number of cores big enough to be competitive with GPUs.

Re:Apple M3 Ultra by kiratp on Wednesday March 05, 2025 @07:16PM (#43274570)
Now compare the FLOPs
Re:Apple M3 Ultra by Rohansi on Wednesday March 05, 2025 @08:33PM (#43275192)
The bandwidth difference likely doesn't make a difference though. Benchmarks of Apple Silicon show that the compute bottlenecks far before running out of bandwidth, even when fully loading all CPU cores, the GPU, etc.
Re:Apple M3 Ultra by Numerlor on Wednesday March 05, 2025 @10:40AM (#43268012)
Ah seems like I remembered the CPU price for a higher tier CPU which can cost the 6k on their own.

Thinking about it you can get a decent 256gb on consumer platforms now too, but the speed will be a bit crap and would need to make sure the platform ully supports ECC UDIMMs

Re:Apple M3 Ultra by behnamoh on Wednesday March 05, 2025 @09:46AM (#43267133)
except that you cannot run multiple language models on Apple Silicon in parallel
Re:Apple M3 Ultra by kevin42 on Wednesday March 05, 2025 @10:20AM (#43267695)
I'm curious why not. I am running a few different models on my mac studio. I'm using llama.cpp, and it performs amazingly fast for the $7k I spent.
Re:Apple M3 Ultra by behnamoh on Wednesday March 05, 2025 @03:55PM (#43272261)
I said in parallel.
Re:Apple M3 Ultra by saagarjha on Thursday March 06, 2025 @02:53AM (#43277589)
Surely you can run smaller models together
Re:Apple M3 Ultra by jeffbee on Wednesday March 05, 2025 @09:47AM (#43267157)
That doesn't sound right. The marginal cost of +768GB of DDR5 ECC memory in an EPYC system is < $5k.
Re:Apple M3 Ultra by InTheArena on Wednesday March 05, 2025 @09:48AM (#43267195)
GPU accessible RAM.
Re:Apple M3 Ultra by numpad0 on Wednesday March 05, 2025 @10:18AM (#43267674)
moot point if tok/s benchmark results are the same or worse.
Re:Apple M3 Ultra by kjreact on Wednesday March 05, 2025 @11:04AM (#43268399)
Are the benchmarks worse? Running LLMs in system memory is rather painful. I am having a hard time finding benchmarks for running large models using system memory. Can you point me to some benchmarks you’re referring to?
Re:Apple M3 Ultra by DrBenCarson on Wednesday March 05, 2025 @10:50AM (#43268201)
Not moot if you care about producing those tokens with the largest available models
Re:Apple M3 Ultra by adrian_b on Wednesday March 05, 2025 @01:07PM (#43270088)
In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher.

Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000.

Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.

Even for computationally-limited programs, the difference between server CPUs and consumer GPUs is not great. One Epyc CPU may have about the same number of FP32 execution units as an RTX 4070, while running at a higher clock frequency (but it lacks the tensor units of an NVIDIA GPU, which can greatly accelerate the execution, where applicable).

Re:Apple M3 Ultra by aurareturn on Wednesday March 05, 2025 @01:40PM (#43270485)

  Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU.
Source on this? CPUs would be very compute constrained.
Re:Apple M3 Ultra by adrian_b on Wednesday March 05, 2025 @03:15PM (#43271721)
According to Apple, the GPU of M3 Ultra has 80 graphics cores, which should mean 10240 FP32 execution units, the same like an NVIDIA RTX 4080 Super.

However Apple does not say anything about the GPU clock frequency, which I assume that it is significantly less than that of NVIDIA.

In comparison, a dual-socket AMD Turin can have up to 12288 FP32 execution units, i.e. 20% more than an Apple GPU.

Moreover, the clock frequency of the AMD CPU must be much higher than that of the Apple GPU, so it is likely that the AMD system may be at least twice faster for computing some graphic application than the Apple M3 Ultra GPU.

I do not know what facilities exist in the Apple GPU for accelerating the computations with low-precision data types, like the tensor cores of NVIDIA GPUs.

While for graphic applications big server CPUs are actually less compute constrained than almost all consumer GPUs (except RTX 4090/5090), the GPUs can be faster for ML/AI applications that use low-precision data types, but this is not at all certain for the Apple GPU.

Even if the Apple GPU happens to be faster for some low-precision data type, the difference cannot be great.

However a server that would beat the Apple M3 Ultra GPU computationally would cost much more than $10k, because it would need CPUs with many cores.

If the goal is only to have a system with 50% more memory and 40% more memory bandwidth than the Apple system, that can be done at a $10k price.

While such a system would become compute constrained more often than an Apple GPU, it would still beat it every time when the memory would be the bottleneck.

Re:Apple M3 Ultra by aurareturn on Wednesday March 05, 2025 @08:55PM (#43275372)
No one is using FP64 for AI inference.
Re:Apple M3 Ultra by adrian_b on Thursday March 06, 2025 @03:30AM (#43277828)
I have not said any word about FP64.

I have just compared the FP32 computational capabilities, i.e. what is used for graphics, between the Apple M3 Ultra GPU and AMD server CPUs, because these numbers are easily available and they demonstrate the size relationships between them.

Both GPUs and server CPUs have greater throughputs for lower precision data (CPUs have instructions for BF16 and INT8 inference), but the exact acceleration factors are hard to find and it is more difficult to estimate the speeds without access to such systems for running benchmarks.

Re:Apple M3 Ultra by sgt on Thursday March 06, 2025 @03:24AM (#43277793)
Anecdotal but it seems like the big EPYC rigs are getting very low tokens per second, and not even consistent. They are strained, as opposed to e.g. M3 Ultra that can likely sustain 40-50 tokens/s based on previous stats.

I'd like to see some proper benchmarking on this though, but it looks like the Apple systems might just be extremely good value if you want to run the large DeepSeek model.

Re:Apple M3 Ultra by energy123 on Wednesday March 05, 2025 @10:23PM (#43276070)
What is the memory bandwidth to the CPU cores? Is it competitive with 8-channel DDR5 servers for non-GPU compute?
Re:Apple M3 Ultra by jeroenhd on Wednesday March 05, 2025 @10:02AM (#43267453)
If you're going to overthrow your entire AI workflow to use a different API anyway, surely the AMD Instinct accelerator cards make more sense. They're expensive, but also a lot faster, and you don't need to deal with making your code work on macOS.
Re:Apple M3 Ultra by wmf on Wednesday March 05, 2025 @12:06PM (#43269323)
Doesn't AMD Instinct cost >$50K for 512GB?
Re:Apple M3 Ultra by codedokode on Wednesday March 05, 2025 @10:09AM (#43267549)
I don't think API has any value because writing software is free and hardware for ML is super expensive.
Re:Apple M3 Ultra by internetter on Wednesday March 05, 2025 @10:15AM (#43267639)
> writing software is free

says who? NVIDIA has essentially entrenched themselves thanks to CUDA

Re:Apple M3 Ultra by knowitnone on Wednesday March 05, 2025 @01:21PM (#43270263)
I'd like to hire you to write free software
Re:Apple M3 Ultra by positr0n on Thursday March 06, 2025 @02:29PM (#43284193)
> writing software is free

Don't tell my boss! I still get paid.

Re:Apple M3 Ultra by baobabKoodaa on Thursday March 06, 2025 @03:38PM (#43284857)
Daniel Ek spotted!
Re:Apple M3 Ultra by chakintosh on Wednesday March 05, 2025 @10:39AM (#43268003)
14k for a maxed out Mac Studio