Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by nathan_compton on Sunday February 22, 2026 @10:03AM
(#47111557)
I'm not sure, but I suspect that LLM weights don't compress all that well. The intuition here is that training an LLM is compression of the training data into the weights, so they are probably very information dense already. Can't squeeze them down much.
[ Parent
| on HN ]
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by spwa4 on Monday February 23, 2026 @02:40AM
(#47119246)
I've found this to often be untrue when optimizing on the CPU. I wish someone would pay me to dive deep into this problem and the scheduling problem. I'd be amazed if I can't squeeze out a 50% speed increase on both problems.