This is also pricing for customers, not what it actually costs to run.

Yes, but that's what's relevant, right? If I create a service, I don't really care if OpenAI makes a killing or is subsidizing my cost with venture capital, what matters to me is how much I pay to OpenAI, and how much revenue my service generates.

It's relevant as far as the providers of said LLM inference could always swoop in, undercut you and take your business if they feel like it. But they could do that anyways, no matter if they do it at a profit or not.

This assumes the service developer adds no value to this process. Because otherwise its like saying nVidia can swoop in on EA's business.

Edit: s/App/service.

Another implication is that you're dependent on it remaining cheap enough. You risk VC money running out and them having to jack up prices, or them doing so because they managed to capture the whole market.

I don’t think it’s cheap because VC money is subsidizing losses on every token. It’s getting cheaper because models and infrastructure are becoming more efficient.

And I really don’t think any of the AI API providers can “capture the whole market”. There are at least 3 of ballpark equal capability, so I don’t see how dramatically raising prices is compatible with dominant market share.

Even if the inference is getting cheaper, all the frontier companies are running massive losses building and serving it. That has to come back eventually, that's just how capitalism works.

Just remember that Netflix didn't start really jacking up the price till after the other players entered the streaming war, when they were pioneers it was dirt cheap. The existence of Disney+ didn't stop them at all.

One major difference is that Netflix has a monopoly on much of its content, but LLMs are fungible.

Dell was never able to jack up its prices, even when it was dominant in the market, because people would just go to another vendor. I think OpenAI is closer to a Dell than a Netflix.

if new models won't require to have significantly better hardware then I think eventually many cases will be run on locally. Currently you have macbook pro with even 128GB RAM - of course almost nobody has it but those hardware will be mainstream common in ~5 years. You already can have ollama - you just need to have equivalent preinstalled by Microsoft/Apple/Google on their all new devices - again most likely this will happen in <5 years.

Haha, maybe Gen AI needs will finally drive up memory needs and the memory installed in most device, but i doubt it.

Apple still happily takes 1800$ for 8GB laptops. Wouldn't be surprised if that were their most sold SKU ans the laptops sold today will be around for quite some time.

Fixed costs vs variable costs. That’s how capitalism works.