News for hackers. Stuff that matters.

A comment on Real-time LLM Inference on Standard GPUs: 3k tokens/s per request. Parent comment

Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by gaeld on Friday May 29, 2026 @11:15AM (#48324176)
thank you deflator, I understand this now! much appreciated
[ Parent | on HN ]
Re:Real-time LLM Inference on Standard GPUs: 3k tokens/s per... by selicos on Friday May 29, 2026 @10:59PM (#48332058)
A consumer "Standard GPU" could mean about a 6-8gb VRAM GPU still in support by the manufacturer, independent of CUDA/etc proprietary technology.

Recent Steam hardware survey top GPU list is:

- RTX 3060 (6 or 12gb VRAM)

- RTX 4060 (8 or 16gb)

- RTX 3050 (6 or 8gb)

- RTX 5070 (12gb)

- RTX 5060 (8gb)

- GTX 1650 (4gb!)

That list only covers about 22% of survey respondents but sets a 6-8gb VRAM baseline for consumer GPUs.

Can this run on an RX 570 8gb form 2017? Maybe that's a ways back. A 1660 6gb from 2019? Intel? They had a decent budget run in recent years.

https://store.steampowered.com/hwsurvey/videocard/

[ Parent | on HN ]