DeepSeek-R1 with Dynamic 1.58-bit Quantization

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by immibis on Tuesday January 28, 2025 @07:03AM (#42851396)

FWIW Threadrippers go up to 1TB and Threadripper Pro up to 2TB. That's even in the lowest model of each series. (I know this because it happens to be the chip I have. Not saying you shouldn't go for Epyc if it works out better.)

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by ant6n on Wednesday January 29, 2025 @05:44AM (#42863556)

Have you tried running the full R1 model with that? People in sibling comments mention high end EPYCs gor a 10K machine, but I’m curious whether it’s possible to make a 1-2K machine that could still run those big models simply because they fit in RAM.

[ Parent | on HN ]

Re:DeepSeek-R1 with Dynamic 1.58-bit Quantization by immibis on Wednesday January 29, 2025 @07:23AM (#42864083)

I spent about $3000 on my machine, have the cheapest Threadripper CPU and 256GB of RAM, so no, 600GB won't fit in RAM on a $2K machine.

But everyone is using the distilled models which are much smaller.

[ Parent | on HN ]