News for hackers. Stuff that matters.
Re:Microsoft introduces its own chips for AI, with eye on cost by pwarner on Wednesday November 15, 2023 @07:01PM (#38284159)
Shouldn't inference be 99.999% of the compute cost over time? Especially for MS. Look how many Copilots they are cramming into their products
Re:Microsoft introduces its own chips for AI, with eye on cost by visarga on Thursday November 16, 2023 @01:16AM (#38286290)
It's probably 25-75 for inference vs training. I estimate 10K tokens/month * 100M users/month = 1T tokens of inference/month. Training is 2-3 times more expensive, the training set is 15T tokens, so about equivalent to 45T tokens of inference. But the lifetime of a model ~1 year, they spend 45T tokens to train and 12T tokens for inference. So the inference cost is about 21% of the total cost.

This estimate is based on a conservative 10K tokens/user/month. In reality it could be much higher if they send PDFs or other input documents. That might increase the ratio maybe to 50-50, but in any estimate it's not 100:1.