News for hackers. Stuff that matters.

A comment on Running DeepSeek R1 Models Locally on NPU. Parent comment

Re:Running DeepSeek R1 Models Locally on NPU by darthrupert on Saturday February 01, 2025 @03:42AM (#42896891)
Wait, what am I running on my 32GB Macbook then? I thought it was the 32b version of deepseek-r1.
[ Parent | on HN ]
Re:Running DeepSeek R1 Models Locally on NPU by RandomBK on Saturday February 01, 2025 @04:03AM (#42896998)
The only 32B distill I'm aware of is `DeepSeek-R1-Distill-Qwen-32B`, which would be a base model of `Qwen-32B` distilled (further trained) on outputs from the full R1 model.
[ Parent | on HN ]
Re:Running DeepSeek R1 Models Locally on NPU by rahimnathwani on Monday February 03, 2025 @01:15AM (#42915468)
That model's weights are around 64GB: https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-...

GP is likely running the 4-bit quantized version of the finetuned Qwen model.

[ Parent | on HN ]
Re:Running DeepSeek R1 Models Locally on NPU by Plankaluel on Saturday February 01, 2025 @03:59AM (#42896978)
You are running Qwen2.5 32b that has been fine tuned on data that was generated by R1
[ Parent | on HN ]
Re:Running DeepSeek R1 Models Locally on NPU by rahimnathwani on Monday February 03, 2025 @01:19AM (#42915490)
Deepseek R1 has 671 billion parameters. Even if you could quantize each parameter to just 1 bit (from 8 bits), you'd still need 84GB of RAM just for the weights. There is no 32B parameter version of the V3/R1 model architecture.
[ Parent | on HN ]