News for hackers. Stuff that matters.

A comment on Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?. Parent comment

Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by brookst on Monday August 11, 2025 @04:50PM (#44869322)
Yes. It is not incredibly rare, it's incredibly common. A huge percentage of queries to retail LLMs are things like "hello" and "what can you do", with static system prompts that make the total context identical.

It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.

[ Parent | on HN ]
Re:Ask HN: How can ChatGPT serve 700M users when I can't run... by throwaway2037 on Thursday August 14, 2025 @02:19AM (#44897276) 

    > It's worth maybe a 3% reduction in GPU usage. So call it a half billion dollars a year or so, for a medium to large service.
So if 3% is 500M, then annual spend is ~16.6B. That is medium sized these days?
[ Parent | on HN ]