> While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.
(Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case)
That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.
Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free.
Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...
It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"
They might not give unfettered credits, it could be for specific projects. That said, I wouldn't be surprised if it was unfettered either.
I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well.
While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).
> ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information.
which, pardon me, but no shit.
Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them from. Especially since OpenAI is definitely not paying list price on those GPU instances.
The other question is how close to capacity their cluster is running, and how much free time on it can be reclaimed, either in terms of spinning down servers for diurnal patterns, or in terms of being able to do training runs in the background.