> ChatGPT could cost OpenAI up to $700,000 a day to run due to "expensive servers," an analyst told The Information.
which, pardon me, but no shit.
Before I break out my back of the envelope calculator, on how many biggest GPU instances in Azure that is, the real question is what their underlying assumptions are, and where they're getting them from. Especially since OpenAI is definitely not paying list price on those GPU instances.
The other question is how close to capacity their cluster is running, and how much free time on it can be reclaimed, either in terms of spinning down servers for diurnal patterns, or in terms of being able to do training runs in the background.
> While training ChatGPT's large language models likely costs tens of millions of dollars, operational expenses, or inference costs, "far exceed training costs when deploying a model at any reasonable scale," Patel and Afzal Ahmad, another analyst at SemiAnalysis, told Forbes. "In fact, the costs to inference ChatGPT exceed the training costs on a weekly basis," they said.
(Looking back, I'm happy that I was careful in my wording in that I didn't say diurnal cycles aren't relevant, just that they aren't as useful in this case)
That said, I suppose I misread the specific suggestion about spinning down servers off-peak and was thinking more about spot pricing at peak vs trough.
Such "free usage" coupons are marketing activities to gain new customers, Microsoft already completed the "dating phase" with OpenAI. They surely don't pay list-price for Azure but it's surely also not free.
Moreover, as per Microsoft themselves, the 1bn USD investment into OpenAI carried the condition that Azure becomes the exclusive provider for cloud-services: https://blogs.microsoft.com/blog/2023/01/23/microsoftandopen...
It's not exclusive because it's free, it's exclusive because "we paid you 1bn USD to buy it from us"
They might not give unfettered credits, it could be for specific projects. That said, I wouldn't be surprised if it was unfettered either.
I'm not sure about the most recent $10 billion investment but I wouldn't be surprised if a significant amount of it is in Azure credits as well.
While that's not "free" (they exchanged equity for it), it's likely not an expense (or at least not an expense that they have to cover fully).
What I am wondering, for those earning 500k, how big is your work load/stress. Would this be a 9-5 job you leave at the office when going home. Or does a job that earns so much consume your life?
I've said it before on here, but I live very comfortably in Philly for a lot less than that.
If interest rates stay elevated, and value investing becomes valuable again, it will be interesting to see how the tech space transforms. When start-ups have to compete with money market funds or even treasuries for investor cash, things become orders of magnitude tighter and more challenging.
I would say 50% the work is harder and consuming and then 50% they can just afford to pay you more and lock up talent because of the wild margins on their products.
Note that there's likely to be some variation per team, but Amazon is famously bad, so ... ;)
Yes, though Switzerland approaches it. If you want to see how much people of various levels of experience get paid at different companies and in different locations go to levels.fyi
Americans get paid much, much more than anyone else.
I am seriously considering a move if my husband can find an academic job over there. The retirement won't be a great lure (fewer years in the system) but we almost have enough to coast from here, so it's about the rest.
I was expecting salaries to cool off a bit with the massive wave of layoffs across the industry, but from what I've seen, that hasn't happened.
~100k€ in (western) Europe may be comparable to ~200k€ in Bay Area.
I'm 20 years into programming and a senior architect and lead on an enterprise project.
I don't even make that first number.
But I value certain things way more than other things, and my current job provides it. Fully remote, leaves me completely alone to accomplish what they need done (and I get it done), unlimited vacation, great benefits, zero pointless meetings (almost an empty calendar).
I'm sure these other companies offer some of that but 500k?! That is absurd.
He sees size as a false measurement of model quality and compares it to the chip speed races we used to see. “I think there’s been way too much focus on parameter count, maybe parameter count will trend up for sure. But this reminds me a lot of the gigahertz race in chips in the 1990s and 2000s, where everybody was trying to point to a big number,” Altman said.
As he points out, today we have much more powerful chips running our iPhones, yet we have no idea for the most part how fast they are, only that they do the job well. “I think it’s important that what we keep the focus on is rapidly increasing capability. And if there’s some reason that parameter count should decrease over time, or we should have multiple models working together, each of which are smaller, we would do that. What we want to deliver to the world is the most capable and useful and safe models. We are not here to jerk ourselves off about parameter count,” he said."
via https://techcrunch.com/2023/04/14/sam-altman-size-of-llms-wo...
What he actually said was that we've reached the point where we can't improve model quality simply by increasing its size (number of parameters). We'll need new techniques to continue to improve.
The company also has about 375 employees. I've no idea how much they get paid but I used $200k as a yearly cost and that comes to $75 million.
That's about 3:1 cost of operating the services to paying employees. That seems quite high as I've never been at a company that had 1:1 costs for running servers vs employee costs but I could entirely be off base here.
Given Sam Altman's recent comments on the days of these LLM being over I think maybe Microsoft or whomever is basically saying that they can't spend that much money and they need to control costs much more heavily.