Re:Apple's MLX adding CUDA support
by albertzeyer on Monday July 14, 2025 @06:59PM
(#44566290)
This is exciting. So this is using unified memory of CUDA? I wonder how well that works. Is the behavior of the unified memory in CUDA actually the same as for Apple silicon? For Apple silicon, as I understand, the memory is anyway shared between GPU and CPU. But for CUDA, this is not the case. So when you have some tensor on CPU, how will it end up on GPU then? This needs a copy somehow. Or is this all hidden by CUDA?
[ Parent
| on HN ]
This is my guess, but does higher end hardware they sell, like the server rack stuff for AI, perhaps have the unified memory?
I know standard GPUs don’t.
The patch suggested one of the reasons for it was to make it easy to develop on a Mac and run on a super computer. So the hardware with the unified memory might be in that class.
The Jetsons[1] have unified memory[2].
[1] https://www.nvidia.com/en-us/autonomous-machines/embedded-sy... [2] https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s...
Re:Apple's MLX adding CUDA support
by tonyarkles on Monday July 14, 2025 @07:56PM
(#44566586)
They sure do and it's pretty amazing. One iteration of a vision system I worked on got frames from a camera over a Mellanox NIC that supports RDMA (Rivermax), preprocessed the images using CUDA, did inference on them with TensorRT, and the first time a single byte of the inference pipeline hit the CPU itself was when we were consuming the output.
Re:Apple's MLX adding CUDA support
by patrickkrusiec on Monday July 14, 2025 @07:19PM
(#44566408)
The physical memory is not be unified, but on modern rack scale Nvidia systems, like Grace Hopper or NVL72, the CPU and the GPU(s) share the same virtual address space and have non-uniform memory access to each other's memory.
The servers don't, but the Jetsons do
Re:Apple's MLX adding CUDA support
by freeone3000 on Monday July 14, 2025 @10:30PM
(#44567318)
Standard GPUs absolutely do. Since CUDA 11, all CUDA cards expose the same featureset at differing speeds (based on backing capability). You can absolutely (try to) run CUDA UMA on your 2060, and it will complete the computation.
In the absence of hardware unified memory, CUDA will automatically copy data between CPU/GPU when there are page faults.
Re:Apple's MLX adding CUDA support
by fenced_load on Monday July 14, 2025 @08:08PM
(#44566633)
There is also NVLink c2c support between Nvidia's CPUs and GPUs that doesn't require any copy, CPUs and GPUs directly access each other's memory over a coherent bus. IIRC, they have 4 CPU + 4 GPU servers already available.
Re:Apple's MLX adding CUDA support
by benreesman on Monday July 14, 2025 @08:35PM
(#44566751)
Yeah NCCL is a whole world and it's not even the only thing involved, but IIRC that's the difference between 8xH100 PCI and 8xH100 SXM2.
Re:Apple's MLX adding CUDA support
by nickysielicki on Monday July 14, 2025 @09:21PM
(#44566987)
This seems like it would be slow…
Re:Apple's MLX adding CUDA support
by freeone3000 on Monday July 14, 2025 @10:28PM
(#44567307)
Matches my experience. It’s memory stalls all over the place, aggravated (on 12.3 at least) there wasn’t even a prefetcher.
Re:Apple's MLX adding CUDA support
by ethan_smith on Tuesday July 15, 2025 @09:40AM
(#44571076)
CUDA's Unified Memory uses page migration with on-demand faulting to create the illusion of shared memory, whereas Apple Silicon has true shared physical memory, resulting in different performance characteristics despite the similar programming model.