News for hackers. Stuff that matters.
Re:ChatGPT spends 'tens of millions of dollars' on 'please' ... by jsnell on Sunday April 20, 2025 @06:42PM (#43747066)
This number does not seem credible. Most likely Altman just made it up.

Back of the envelope:

OpenAI inference costs last year were 4B. Tens of millions would be at least 20M, i.e. 0.5%.

That 4B is not just the electricity cost. It needs to cover the amortized cost of the hardware, the cloud provider's margin, etc.

Let's say a H100 costs $30k, and has a lifetime of 5 years. I make that about $16 / day in depreciation. The H100 run at 100% utilization will use 17kWH of electricity in a day. What does that cost? $2-$3 / day? Let's assume the cloud provider's margin is 0. That still means power consumption is maybe 1/5th of the total inference cost.

So the comparison is 800M vs 20M (2.5%).

Can 2.5% of their tokens be pleasantries? Seems impossible. A "please" is a single token, which will be totally swamped by the output, which will typically be 1000x that.

Re:ChatGPT spends 'tens of millions of dollars' on 'please' ... by shonenknifefan1 on Sunday April 20, 2025 @09:55PM (#43747910)
A 'thank you' that might otherwise end a conversation will trigger a response by the model that will generate additional tokens. So the inclusion of the pleasantries could increase tokens beyond the pleasantries themselves.
Re:ChatGPT spends 'tens of millions of dollars' on 'please' ... by pixelesque on Sunday April 20, 2025 @10:11PM (#43747983)
Saying "thank you" as a response to an answer from both ChatGPT and Claude generally involves a follow-up response from the LLM, sometimes prompting if you'd like further information, so it's obviously going to involve some cost of that additional "final" response from the user, and then a follow-up from the LLM itself in terms of parsing and inference.