Re:DeepSeek-R1
by ein0p on Tuesday January 21, 2025 @02:40PM
(#42784370)
My models are both 4 bit. But yeah, that could be - small models are much worse at tolerating quantization. That's why people use LoRA to recover the accuracy somewhat even if they don't need domain adaptation.
It could be the quantized version failing?