Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by xaskasdf on Saturday February 21, 2026 @08:11PM
(#47107002)
yeah, actually I wanted to see if this was possible at all. I managed to get around 3000 tokens/s on a ps2 with classic transformers, since the emotion engine is capable of 32 bit addresses, but it has like 32gb of ram. So I ran into the question of why was that fast and I couldn't get that speed even with small models, and the deal is that the instructions went right of the memory to the gpu and that's the main difference that does when a regular computer does inference: it has to request the instructions to the cpu every time. As I mentioned too, on professional cards you can avoid these problems naturally, since they got instructions precisely for this, but sadly I don't have 30k bucks to spare on a gpu :(
[ Parent
| on HN ]
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by derstander on Saturday February 21, 2026 @08:32PM
(#47107152)
*32MB of RAM (plus 4MB of video RAM and a little sound and IOP memory).
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by anoncow on Saturday February 21, 2026 @11:14PM
(#47108082)
3000 tokens per sec on 32 mb Ram?
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by fc417fc802 on Saturday February 21, 2026 @11:56PM
(#47108321)
fast != practical
You can get lots of tokens per second on the CPU if the entire network fits in L1 cache. Unfortunately the sub 64 kiB model segment isn't looking so hot.
But actually ... 3000? Did GP misplace one or two zeros there?
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by xaskasdf on Sunday February 22, 2026 @10:48AM
(#47111944)
I wondered the same, but the rendering seems right, the output was almost instant. I'll recheck the token counter; anyway as you say, fast isn't practical. Actually I had to develop my own tiny model https://huggingface.co/xaskasdf/brandon-tiny-10m-instruct to fit something "usable", and it's basically a liar or disinformation machine haha
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by eleventyseven on Sunday February 22, 2026 @12:46AM
(#47108562)
> I don't have 30k bucks to spare on a gpu :(
Do you have $2/hr to rent an RTX 6000 96GB or $5/hr for B200 180GB on the cloud?
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by superkuh on Sunday February 22, 2026 @12:47AM
(#47108575)
I'd rather not give money to scalper barons if I can avoid it. Fab capacity is going to that for rental rather than hardware for humans.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by xaskasdf on Sunday February 22, 2026 @10:00AM
(#47111512)
I thought about that, but idk if they allow me to modify the linux kernel and nvidia cuda kernel at all
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by green-salt on Sunday February 22, 2026 @12:23PM
(#47112779)
I think you can do a bunch of that on Digitalocean's GPU droplets.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by jonassm on Sunday February 22, 2026 @12:37PM
(#47112889)
In those systems you could probably leverage something like Nvidia SCADA or GDS directly.
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by xaskasdf on Sunday February 22, 2026 @03:07PM
(#47114159)
Actually since they have direct GDS it should perform really well on professional gpus
Re:Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-G...
by SmithRenaldo on Monday February 23, 2026 @07:07PM
(#47130947)
The $5/hr B200 rate is fine for training, but cloud latency usually breaks real-time signal processing. I’ve been hitting similar walls with MemeRadar; when you're processing high-volume spikes, the bottleneck is memory bandwidth, not raw TFLOPS. Quantizing to fit L3 cache is an option, but you lose the precision needed for spotting subtle rug-pull patterns. For 24/7 production workloads, local hardware TCO usually beats cloud rentals.