News for hackers. Stuff that matters.
Re:Deepseek R1-0528 by danielhanchen on Wednesday May 28, 2025 @09:14PM (#44122138)
Oh you can still run them unquantized! See https://docs.unsloth.ai/basics/llama-4-how-to-run-and-fine-t... where we show you can offload all MoE layers to system RAM, and leave non MoE layers on the GPU - the speed is still pretty good!

You can do it via `-ot ".ffn_.*_exps.=CPU"`

Re:Deepseek R1-0528 by behnamoh on Wednesday May 28, 2025 @11:09PM (#44122757)
Thanks, I'll try it! I guess "mixing" GPU+CPU would hurt the perf tho.