The other thing to note is that OpenAI limits paying users to X queries per Y hours in GPT-4, something like 3 per 5. Which means the GPT-4 fleet is quite limited.
It also keeps saying "expect significantly lower caps, as we adjust for demand", but it's been three weeks now and they haven't lowered it yet.
> I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization,
I don't think you've thought this through. There's no realistic level of utilization that's going to make up for 5 orders of magnitude. An unimaginably crappy 1% utilization would still leave you off by 3 orders of magnitude.
This very submission about OpenAI having to suspend selling subscriptions makes it pretty clear they're not having trouble with low utilization...
> If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high.
You do understand that this doesn't actually support your initial estimate, right? It just makes it more obvious how ridiculous it was.
I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl...
I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries.
If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.