News for hackers. Stuff that matters.

A comment on Apple M3 Ultra. Parent comment

Re:Apple M3 Ultra by atwrk on Wednesday March 05, 2025 @01:25PM (#43270310)
If I'm not mistaken, each token produced roughly equals the whole model in memory transfers (the exception being MoE models). That's why memory bandwidth is so important in the first place, or not?
[ Parent | on HN ]
Re:Apple M3 Ultra by wmf on Wednesday March 05, 2025 @04:12PM (#43272492)
My understanding is that if you can store 1/Nth of the weights in RAM on each of the N nodes then there's no need to send the weights over the network.
[ Parent | on HN ]
Re:Apple M3 Ultra by unsatchmo on Thursday March 06, 2025 @11:37AM (#43282138)
You're correct about the weights: each machine could in fact store all of the weights. However I think you still have to transfer the activations and the KV-Cache while performing inference.
[ Parent | on HN ]