Re:Microsoft introduces its own chips for AI, with eye on cost
by pwarner on Wednesday November 15, 2023 @07:01PM
(#38284159)
Shouldn't inference be 99.999% of the compute cost over time?
Especially for MS.
Look how many Copilots they are cramming into their products
[ Parent
| on HN ]
Re:Microsoft introduces its own chips for AI, with eye on cost
by visarga on Thursday November 16, 2023 @01:16AM
(#38286290)
It's probably 25-75 for inference vs training. I estimate 10K tokens/month * 100M users/month = 1T tokens of inference/month. Training is 2-3 times more expensive, the training set is 15T tokens, so about equivalent to 45T tokens of inference. But the lifetime of a model ~1 year, they spend 45T tokens to train and 12T tokens for inference. So the inference cost is about 21% of the total cost.
This estimate is based on a conservative 10K tokens/user/month. In reality it could be much higher if they send PDFs or other input documents. That might increase the ratio maybe to 50-50, but in any estimate it's not 100:1.