Apple M2 Ultra Isn't Faster Than Last Gen AMD&Intel Desktop CPUs, 50% Slower GPU

Re:Apple M2 Ultra Isn't Faster Than Last Gen AMD&Intel Deskt... by samwillis on Saturday June 10, 2023 @03:26PM (#36274118)

I think the exciting thing with Apple silicone and "AI" is inference, not training. Due to their unified memory, you can potentially have enormous local model inference. That's potentially much more expensive on a PC with a GPU having its own memory.

Apple have an opportunity, if they 2-4x the memory on the entry level devices (not beyond the realms of possibility), make local inference a thing available to all.

[ Parent | on HN ]

Re:Apple M2 Ultra Isn't Faster Than Last Gen AMD&Intel Deskt... by danieldk on Saturday June 10, 2023 @05:04PM (#36275087)

Sure, but for inference it’s much easier to work in lower precision. Training with very low precision is also possible, but is often more tricky due to numerical stability.

A lot of work is going on in 8-bit inference and even 4 bit inference. So, models that need 64 GB in FP32 can do with 16GB VRAM in FP8 or INT8, which is well within the realm of consumer NVIDIA cards. And the latest NVIDIA tensor cores will absolutely destroy Apple Silicon GPUs or the Neural Engine in 8 bit.

So, I don’t think it’s really a strong argument. And as someone who is a Mac user and a ML practitioner, I’d be very happy if they started supporting eGPUs again.

Apple Silicon has many strengths and the GPU core are fine for many ends, from games to graphics apps.

But let’s not pretend that Apple is beating NVIDIA at their own game (yet). That day might come, but currently it only leads to disappointed users in ML forums who were hyped into thinking that their vanilla M2 MacBook Airs can almost compete with a 4090 in training a deep transformer model. (Yes, that happens.)

[ Parent | on HN ]

Re:Apple M2 Ultra Isn't Faster Than Last Gen AMD&Intel Deskt... by incrudible on Saturday June 10, 2023 @07:36PM (#36276461)

Multiple NVIDIA GPUs also have unified memory from an application perspective, one GPU has up to 80GBs right now. They also have a proper software stack. If NVIDIA saw any credible threat there, they could ramp that up, DRAM is not beholden to Apple.

[ Parent | on HN ]