Re:Microsoft introduces its own chips for AI, with eye on cost
by visarga on Thursday November 16, 2023 @01:16AM
(#38286290)
It's probably 25-75 for inference vs training. I estimate 10K tokens/month * 100M users/month = 1T tokens of inference/month. Training is 2-3 times more expensive, the training set is 15T tokens, so about equivalent to 45T tokens of inference. But the lifetime of a model ~1 year, they spend 45T tokens to train and 12T tokens for inference. So the inference cost is about 21% of the total cost.
This estimate is based on a conservative 10K tokens/user/month. In reality it could be much higher if they send PDFs or other input documents. That might increase the ratio maybe to 50-50, but in any estimate it's not 100:1.
This estimate is based on a conservative 10K tokens/user/month. In reality it could be much higher if they send PDFs or other input documents. That might increase the ratio maybe to 50-50, but in any estimate it's not 100:1.