DeepSeek-R1 with Dynamic 1.58-bit Quantization

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by smcleod on Tuesday January 28, 2025 @07:01AM (#42851389)

I just ran it up on 48gb (2x 3090) + overflow into CPU RAM and it runs at around 4tk/s (only a little 8k context size though) which while absolutely not something I'd personally use daily - it is actually usable.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by eurekin on Tuesday January 28, 2025 @09:01AM (#42852376)

I have similar set-up - can you help out with running it? Was it in ollama?

EDIT: It seems that original authors provided a nice write-up:

https://unsloth.ai/blog/deepseekr1-dynamic#:~:text=%F0%9F%96...

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by smcleod on Tuesday January 28, 2025 @02:39PM (#42856917)

Yep that's pretty much what I did, their calculation for the layers was slightly off though, I found I could offload an extra 1-2 layers to the GPUs

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @04:27PM (#42858263)

Oh yes I reduced it by 4 for just in case :) I found sometimes the formula doesn't work, so in the worst case -4 was used - glad at least it ran!

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by ryao on Tuesday January 28, 2025 @03:44PM (#42857718)

Which one did you run?

[ Parent | on HN ]