Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by yorwba on Tuesday January 28, 2025 @05:43AM
(#42850897)
Each MoE layer has its own router, and it activates 8 (out of 256) experts at a time. There's no reason to expect all of them to stay on the same GPU, so you're pretty much guaranteed to have to do all-to-all communication between the GPUs in your cluster after every layer for every token.
[ Parent
| on HN ]
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by phire on Tuesday January 28, 2025 @07:01AM
(#42851388)
Interesting.
I had assumed the performance advantage for MoE came from minimising traffic between GPUs. But if it's per layer routing, then it's going to massively increase inter-gpu traffic compared to vertical slicing.
I guess that means the performance advantage actually comes when batching thousands of queries? The MoE routing would mean that on each MoE layer, each GPU shard gets a batch of queries that will all hit roughly the same subset of experts (and read the same weights from memory). The batches then shuffle between each MoE layer to re-optimise.
It's kind of like GPU raytracing where you get large performance gains by running coherency sorting on rays and batching similar rays together.
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by yorwba on Tuesday January 28, 2025 @09:08AM
(#42852448)
The performance advantage comes from doing 1/32 of the floating point operations compared to a dense layer with the same number of parameters.
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by iamnotagenius on Tuesday January 28, 2025 @09:22AM
(#42852597)
The performance comes mostly from a fraction of memory bandwidth needed, as LLM are mostly memory constrained. Compute matters too, but usually far less than memory.