News for hackers. Stuff that matters.
Re:Show HN: Can I run this LLM? (locally) by monocasa on Saturday March 08, 2025 @07:52PM (#43305082)
Is it maybe because DeepSeek is a MoE and doesn't require all parameters for a given token?

That's not ideal from a token throughput perspective, but I can see min working set of weight memory gains if you can load pieces into vram for each token.

Re:Show HN: Can I run this LLM? (locally) by throwaway314155 on Saturday March 08, 2025 @07:58PM (#43305129)
It still wouldn't fit in 16 GB memory. Further there's too much swapping going on with MoE models to move expert layers to and from gpu without bottlenecks.