Re:Show HN: Can I run this LLM? (locally)
by throwaway314155 on Saturday March 08, 2025 @07:58PM
(#43305129)
It still wouldn't fit in 16 GB memory. Further there's too much swapping going on with MoE models to move expert layers to and from gpu without bottlenecks.
That's not ideal from a token throughput perspective, but I can see min working set of weight memory gains if you can load pieces into vram for each token.