They are talking about training models, though. Run is a bit ambiguous, is that also what you mean?

No.

For training the Macs do have some interesting advantages due to the unified memory. The GPU cores have access to all of system RAM (and also the system RAM is ridiculously fast - 400GB/sec when DDR4 is barely 30GB/sec, which has a lot of little fringe benefits of it's own, part of why the Studio feels like an even more powerful machine than it actually is. It's just super snappy and responsive, even under heavy load.)

The largest consumer NVidia card has 22GB of useable RAM.

The $1999 Mac has 32GB, and for $400 more you get 64GB.

$3200 gets you 96GB, and more GPU cores. You can hit the system max of 192GB for $5500 on an Ultra, albeit it with the lessor GPU.

Even the recently announced 6000-series AI-oriented NVidia cards max out at 48GB.

My understanding is a that a lot of enthusiasts are using Macs for training because for certain things having more RAM is just enabling.

The huge amount of optimizations available on Nvidia and not available on Apple make the reduced VRAM worth it, because even the most bloated of foundation models will have some magical 0.1bit quantization technique be invented by a turbo-nerd which only works on Nvidia.

I keep hearing this meme of Mac's being a big deal in LLM training, but I have seen zero evidence of it, and I am deeply immersed in the world of LLM training, including training from scratch.

Stop trying to meme apple M chips as AI accelerators. I'll believe it when unsloth starts to support a single non-nvidia chip.

If you work for a company willing to shell out sure there are better options.

But for individual developers it’s an interesting proposition.

And a bigger question is: what if you already have (or were going to buy) a Mac? You prefer them or maybe are developing for Apple platforms.

Upping the chip or memory could easily be cheaper than getting a PC rig that’s faster for training. That may be worth it to you.

Not everyone is starting from zero or wants the fastest possible performance money can buy ignoring all other factors.

Agreed. Although inference is good enough on the Mac, there is no way I am training on them at all.

It's just more efficient to offload training to cloud Nvidia GPUs

Yeah, and I think people forget all the time that inference (usually batch_size=1) is memory bandwidth bound, but training (usually batch_size=large) is usually compute bound. And people use enormous batch sizes for training.

And while the Mac Studio has a lot of memory bandwidth compared to most desktops CPUs, it isn't comparable to consumer GPUs (the 3090 has a bandwidth of ~936GBps) let alone those with HBM.

I really don't hear about anyone training on anything besides NVIDIA GPUs. There are too many useful features like mixed-precision training, and don't even get me started on software issues.