News for hackers. Stuff that matters.
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @05:38AM (#42850869)
Oh yes 192GB machines should be able these quants (131GB for 1.58bit, 158GB for 1.73bit, 183GB for 2.22bit) well :)
Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by bradfox2 on Tuesday January 28, 2025 @08:24AM (#42852014)
Great release Daniel. Applaud the consistency you have shown.

Can you release slightly bigger quant versions? Would enjoy something that runs well on 8x32 v100 and 8x80 A100.

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by danielhanchen on Tuesday January 28, 2025 @04:31PM (#42858314)
Thanks! Oh I did release 4bit quants, 5bit, 6bit etc all at https://huggingface.co/unsloth/DeepSeek-R1-GGUF if that helps - they're not dynamic though but it should function fine :)