DeepSeek-R1 with Dynamic 1.58-bit Quantization

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by bradfox2 on Tuesday January 28, 2025 @08:24AM (#42852014)

Great release Daniel. Applaud the consistency you have shown.

Can you release slightly bigger quant versions? Would enjoy something that runs well on 8x32 v100 and 8x80 A100.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @04:31PM (#42858314)

Thanks! Oh I did release 4bit quants, 5bit, 6bit etc all at https://huggingface.co/unsloth/DeepSeek-R1-GGUF if that helps - they're not dynamic though but it should function fine :)

[ Parent | on HN ]