News for hackers. Stuff that matters.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by zozbot234 on Saturday February 21, 2026 @07:28PM (#47106680)
Should be active param size, not model size.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by tgrowazay on Monday February 23, 2026 @02:55PM (#47127835)
Yes, you’re right.

LLama 3.1 however is not MoE, so all params are active.

For MoE it is tricky, because for each token you only use a subset of params (an “expert”) but you don’t know which one, so you have to keep them all in memory or wait until it loads from slower storage, potentially different for each token.