DeepSeek-R1 with Dynamic 1.58-bit Quantization

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by eurekin on Tuesday January 28, 2025 @09:01AM (#42852376)

I have similar set-up - can you help out with running it? Was it in ollama?

EDIT: It seems that original authors provided a nice write-up:

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by smcleod on Tuesday January 28, 2025 @02:39PM (#42856917)

Yep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @04:27PM (#42858263)

Oh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!

[ Parent | on HN ]