News for hackers. Stuff that matters.
Re:ChatGPT could cost over $700k per day to operate by vitus on Friday April 21, 2023 @08:50AM (#35652929)
The estimate in the article pins most of the cost on inference, not training, so diurnal patterns are unfortunately not as useful here.

> While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.

Re:ChatGPT could cost over $700k per day to operate by zamnos on Friday April 21, 2023 @09:13AM (#35653222)
Why wouldn't inference follow diurnal patterns?
Re:ChatGPT could cost over $700k per day to operate by reaperman on Friday April 21, 2023 @10:12AM (#35654044)
Agreed. That seems backwards. Training would not follow circadian rhythms, inference would.
Re:ChatGPT could cost over $700k per day to operate by vitus on Friday April 21, 2023 @10:51AM (#35654633)
I should clarify: training is not latency sensitive, so you can run your workloads at off-peak hours when people are asleep. Inference means you need to run your workloads at peak when people are using your service.

(Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case)

That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.