DeepSeek-R1 with Dynamic 1.58-bit Quantization

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @05:41AM (#42850888)

Hey! :) Coincidentally the seeds I always use are 3407, 3408 and 3409 :) 3407 because of https://arxiv.org/abs/2109.08203

I also tried not setting the seeds, but the results are still the same - quantizing all layers seems to make the model forget and repeat everything - I put all examples here: https://docs.unsloth.ai/basics/deepseek-r1-dynamic-1.58-bit#...

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by iamnotagenius on Tuesday January 28, 2025 @09:13AM (#42852488)

would be great to have dynamic quants of V3-non-R1 version, as for some tasks it is good enough. Also would be very interesting to see degradation with dynamic quants on small/medium size MoEs, such as older Deepseek models, Mixtrals, IBM tiny Granite MoE. Would be fun if Granite 1b MoE will still be functioning at 1.58bit.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @04:32PM (#42858329)

Oh yes multiple people have asked me about this - I'll see what I can do :)

[ Parent | on HN ]