News for hackers. Stuff that matters.
Re:Deepseek R1-0528 by danielhanchen on Wednesday May 28, 2025 @03:05PM (#44119514)
We made DeepSeek R1 run on a local device via offloading and 1.58bit quantization :) https://unsloth.ai/blog/deepseekr1-dynamic

I'm working on the new one!

Re:Deepseek R1-0528 by screaminghawk on Wednesday May 28, 2025 @03:37PM (#44119884)
I use this a lot! Thanks for your work and looking forward to the next one
Re:Deepseek R1-0528 by danielhanchen on Wednesday May 28, 2025 @03:43PM (#44119949)
Thank you!! New versions should be much better!
Re:Deepseek R1-0528 by CamperBob2 on Wednesday May 28, 2025 @03:41PM (#44119931)
Your 1.58-bit dynamic quant model is a religious experience, even at one or two tokens per second (which is what I get on my 128 MB Raptor Lake+4090). It's like owning your own genie... just ridiculously smart. Thanks for the work you've put into it!
Re:Deepseek R1-0528 by danielhanchen on Wednesday May 28, 2025 @03:43PM (#44119947)
Oh thank you! :) Glad they were useful!
Re:Deepseek R1-0528 by nxobject on Wednesday May 28, 2025 @07:55PM (#44121670)
Likewise - for me, it feels how I imagined getting a microcomputer in the 70s was like.
Re:Deepseek R1-0528 by danielhanchen on Wednesday May 28, 2025 @09:14PM (#44122144)
:) The good ol days!
Re:Deepseek R1-0528 by behnamoh on Wednesday May 28, 2025 @08:08PM (#44121744)
> 1.58bit quantization

of course we can run any model if quantize it enough. but I think the OP was talking about the unquantized version.

Re:Deepseek R1-0528 by danielhanchen on Wednesday May 28, 2025 @09:14PM (#44122138)
Oh you can still run them unquantized! See https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-t... where we show you can offload all MoE layers to system RAM, and leave non MoE layers on the GPU - the speed is still pretty good!

You can do it via `-ot ".ffn_.*_exps.=CPU"`

Re:Deepseek R1-0528 by behnamoh on Wednesday May 28, 2025 @11:09PM (#44122757)
Thanks, I'll try it! I guess "mixing" GPU+CPU would hurt the perf tho.