News for hackers. Stuff that matters.

A comment on Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?. Parent comment

Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by oceanplexian on Friday August 08, 2025 @07:37PM (#44842797)
I can’t speak to the Tesla stuff but I run an Epyc 7713 with a single 3090 and creatively splitting the model between GPU/8 channels of DDR4 I can do about 9 tokens per second on a q4 quant.
[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by CamperBob2 on Friday August 08, 2025 @07:44PM (#44842836)
Impressive. Is that a distillation, or the real thing?
[ Parent | on HN ]