Apple M3 Ultra - Hackdot

Re:Apple M3 Ultra by atwrk on Wednesday March 05, 2025 @11:14AM (#43268547)

But 80Gbit/s is way slower than even regular dual channel RAM, or am I missing something here? That would mean the LLM would be excruciatingly slow. You could get an old EPYC for a fraction of that price and have more performance.

[ Parent | on HN ]

Re:Apple M3 Ultra by wmf on Wednesday March 05, 2025 @11:59AM (#43269233)

The weights don't go over the network so performance is OK.

[ Parent | on HN ]

Re:Apple M3 Ultra by atwrk on Wednesday March 05, 2025 @01:25PM (#43270310)

If I'm not mistaken, each token produced roughly equals the whole model in memory transfers (the exception being MoE models). That's why memory bandwidth is so important in the first place, or not?

[ Parent | on HN ]

Re:Apple M3 Ultra by wmf on Wednesday March 05, 2025 @04:12PM (#43272492)

My understanding is that if you can store 1/Nth of the weights in RAM on each of the N nodes then there's no need to send the weights over the network.

[ Parent | on HN ]

Re:Apple M3 Ultra by unsatchmo on Thursday March 06, 2025 @11:37AM (#43282138)

You're correct about the weights: each machine could in fact store all of the weights. However I think you still have to transfer the activations and the KV-Cache while performing inference.

[ Parent | on HN ]