Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per...
by deflator on Friday May 29, 2026 @10:10AM
(#48323326)
What a lot of use on here are salivating for is the ability to run these on prosumer hardware at home. So we tend to jump to the conclusion that "standard" means "consumer-grade" because that's what we want to see.
Still, very cool work!
[ Parent
| on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per...
by gaeld on Friday May 29, 2026 @11:15AM
(#48324176)
thank you deflator, I understand this now! much appreciated
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per...
by selicos on Friday May 29, 2026 @10:59PM
(#48332058)
A consumer "Standard GPU" could mean about a 6-8gb VRAM GPU still in support by the manufacturer, independent of CUDA/etc proprietary technology.
Recent Steam hardware survey top GPU list is:
- RTX 3060 (6 or 12gb VRAM)
- RTX 4060 (8 or 16gb)
- RTX 3050 (6 or 8gb)
- RTX 5070 (12gb)
- RTX 5060 (8gb)
- GTX 1650 (4gb!)
That list only covers about 22% of survey respondents but sets a 6-8gb VRAM baseline for consumer GPUs.
Can this run on an RX 570 8gb form 2017? Maybe that's a ways back. A 1660 6gb from 2019? Intel? They had a decent budget run in recent years.