News for hackers. Stuff that matters.

A comment on Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?. Parent comment

Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by spott on Monday August 11, 2025 @10:49AM (#44864772)
KV cache for dense models is order 50% of parameters. For sparse moe models it can be significantly smaller I believe, but I don’t think it is measured in kb.
[ Parent | on HN ]

There are no comments on this comment yet.