I’ve created a framework that lets me test the quality in automated way between prompt changes and models and I compare costs/speed/quality.
The only thing that requires humans to judge the qualify out of all those are RAG results.
One of Anthropics models did the best with image understanding with Amazon’s Nova Pro being slightly behind.
For my tests, I used a customer’s specific set of test data.
For RAG I forgot. But is much more subjective. I just gave the customer an ability to configure the model and modify the prompt so they could choose.
But, talk to any (or almost any) non-developer and you'll find they 1/ mostly only use ChatGPT, sometimes only know of ChatGPT and have never heard of any other solution, and 2/ in the rare case they did switch to something else, they don't want to go back, they're gone for good.
Each provider has a moat that is its number of daily users; and although it's a little annoying to admit, OpenAI has the biggest moat of them all.
I would think that Gemini (the model) will add profit to Google way before OpenAI ever becomes profitable as they leverage it within their business.
Why would I pay for openrouter.ai and add another dependency? If I’m just using Amazon Bedrock hosted models, I can just use the AWS SDK and change the request format slightly based on the model family and abstract that into my library.
https://www.ft.com/content/fce77ba4-6231-4920-9e99-693a6c38e...
I think the combination of AI overviews and a separate “AI mode” tab is good enough.
https://docs.aws.amazon.com/code-library/latest/ug/python_3_...
Every model family has its own request format.
When I said it was “trivial” to write a library, I should have been more honest. “It’s trivial to point ChatGPT to the documentation and have it one shot creating a Python library for the models you want to support”.
That said, I don't believe oai's models consistently produce the best results.
maybe another way of saying the same thing is that there is still a lot of work to make eval tooling a lot better!
Theres too much entropy in the system. Context babysitting is our future.
It wasn't a huge lift, but there is some moat. And the results were worse than for GPT-5 which I suppose is not a surprise, it was always unlikely GPT-5 was wasting all those flops.
It’s extremely easy to write a library that makes switching between models trivial. I could add OpenAI support. It would be just slightly more complicated because I would have to have a separate set of API keys while now I can just use my AWS credentials.
Also of course latency would be theoretically worse since with hosting on AWS and using AWS for inference you stay within the internal network (yes I know to use VPC endpoints).
There is no moat around switching models unlike Ben says.