News for hackers. Stuff that matters.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by spwa4 on Sunday February 22, 2026 @05:16AM (#47109848)
I've often wondered doing this with extreme compression. What if you did extreme compression + decompression on the GPU? Because you're leaving a lot of compute unused.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by xaskasdf on Sunday February 22, 2026 @09:42AM (#47111381)
I did it, but with different quantization compressions, It ran into quality issues, I will try to rerun with the same quants if that fixes the issue, but the most that looks unused, its being used by rotating layers that are being swapped by the cpu from the ram itself, that manages to keep layers warm, ready to use while inferencing and discarding already used ones
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by nathan_compton on Sunday February 22, 2026 @10:03AM (#47111557)
I'm not sure, but I suspect that LLM weights don't compress all that well. The intuition here is that training an LLM is compression of the training data into the weights, so they are probably very information dense already. Can't squeeze them down much.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by spwa4 on Monday February 23, 2026 @02:40AM (#47119246)
I've found this to often be untrue when optimizing on the CPU. I wish someone would pay me to dive deep into this problem and the scheduling problem. I'd be amazed if I can't squeeze out a 50% speed increase on both problems.