Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by smcleod on Tuesday January 28, 2025 @02:39PM
(#42856917)
Yep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs
[ Parent
| on HN ]
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization
by danielhanchen on Tuesday January 28, 2025 @04:27PM
(#42858263)
Oh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!