Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by yorwba on Tuesday January 28, 2025 @09:08AM
(#42852448)
The performance advantage comes from doing 1/32 of the floating point operations compared to a dense layer with the same number of parameters.
[ Parent
| on HN ]
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by iamnotagenius on Tuesday January 28, 2025 @09:22AM
(#42852597)
The performance comes mostly from a fraction of memory bandwidth needed, as LLM are mostly memory constrained. Compute matters too, but usually far less than memory.