News for hackers. Stuff that matters.

A comment on Apple introduces M4 chip. Parent comment

Re:Apple introduces M4 chip by Der_Einzige on Tuesday May 07, 2024 @09:26PM (#40293438)
The huge amount of optimizations available on Nvidia and not available on Apple make the reduced VRAM worth it, because even the most bloated of foundation models will have some magical 0.1bit quantization technique be invented by a turbo-nerd which only works on Nvidia.

I keep hearing this meme of Mac's being a big deal in LLM training, but I have seen zero evidence of it, and I am deeply immersed in the world of LLM training, including training from scratch.

Stop trying to meme apple M chips as AI accelerators. I'll believe it when unsloth starts to support a single non-nvidia chip.

[ Parent | on HN ]
Re:Apple introduces M4 chip by MBCook on Tuesday May 07, 2024 @11:43PM (#40294117)
If you work for a company willing to shell out sure there are better options.

But for individual developers it’s an interesting proposition.

And a bigger question is: what if you already have (or were going to buy) a Mac? You prefer them or maybe are developing for Apple platforms.

Upping the chip or memory could easily be cheaper than getting a PC rig that’s faster for training. That may be worth it to you.

Not everyone is starting from zero or wants the fastest possible performance money can buy ignoring all other factors.

[ Parent | on HN ]
Re:Apple introduces M4 chip by arvinsim on Wednesday May 08, 2024 @02:25AM (#40294941)
Agreed. Although inference is good enough on the Mac, there is no way I am training on them at all.

It's just more efficient to offload training to cloud Nvidia GPUs

[ Parent | on HN ]
Re:Apple introduces M4 chip by singhrac on Wednesday May 08, 2024 @03:52AM (#40295446)
Yeah, and I think people forget all the time that inference (usually batch_size=1) is memory bandwidth bound, but training (usually batch_size=large) is usually compute bound. And people use enormous batch sizes for training.

And while the Mac Studio has a lot of memory bandwidth compared to most desktops CPUs, it isn't comparable to consumer GPUs (the 3090 has a bandwidth of ~936GBps) let alone those with HBM.

I really don't hear about anyone training on anything besides NVIDIA GPUs. There are too many useful features like mixed-precision training, and don't even get me started on software issues.

[ Parent | on HN ]