News for hackers. Stuff that matters.

A comment on Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?. Parent comment

Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by CamperBob2 on Friday August 08, 2025 @05:30PM (#44841884)
Are you talking about the guy in Temecula running two different auctions with some of the same photos (356878140643 and 357146508609, both showing a missing heat sink?) Interesting, but seems sketchy.

How useful is this Tesla-era hardware on current workloads? If you tried to run the full DeepSeek R1 model on it at (say) 4-bit quantization, any idea what kind of TTFT and TPS figures might be expected?

[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by oceanplexian on Friday August 08, 2025 @07:37PM (#44842797)
I can’t speak to the Tesla stuff but I run an Epyc 7713 with a single 3090 and creatively splitting the model between GPU/8 channels of DDR4 I can do about 9 tokens per second on a q4 quant.
[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by CamperBob2 on Friday August 08, 2025 @07:44PM (#44842836)
Impressive. Is that a distillation, or the real thing?
[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by justincormack on Saturday August 09, 2025 @09:00AM (#44846156)
Tesla doesnt support 4 bit float.
[ Parent | on HN ]