News for hackers. Stuff that matters.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by fc417fc802 on Saturday February 21, 2026 @11:56PM (#47108321)
fast != practical

You can get lots of tokens per second on the CPU if the entire network fits in L1 cache. Unfortunately the sub 64 kiB model segment isn't looking so hot.

But actually ... 3000? Did GP misplace one or two zeros there?

Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G... by xaskasdf on Sunday February 22, 2026 @10:48AM (#47111944)
I wondered the same, but the rendering seems right, the output was almost instant. I'll recheck the token counter; anyway as you say, fast isn't practical. Actually I had to develop my own tiny model https://huggingface.co/xaskasdf/brandon-tiny-10m-instruct to fit something "usable", and it's basically a liar or disinformation machine haha