> On top of all the other things that add uncertainty to our software (flaky servers, third-party APIs going down, scaling issues),

Plus the elephant in the room, that prompt injection and hallucinations are a fundamental way of life for the algorithm, even when it's not getting updated in the background.

Yep, that's also a huge issue with LLMs in production. Our product has some automatic detection of jailbreak attempts so that you can see when people are trying to jailbreak & prompt inject, but hallucinations are the biggest unsolved problem imo.

People have been complaining about AI models surreptitiously changing underneath them for a while now, and we found evidence of it happening in the wild. We build an LLM monitoring and testing tool called Libretto, and we saw GPT-4o start to behave significantly differently on one of our prompts this week. This is a write-up of how we detected the change and what it means for building on top of LLMs that can change at any moment.

Just goes to show how important having control over the model is for actual production applications

Yeah, that's definitely another way to solve the issue. Of course, that can add a ton of operational complexity and means that you are responsible for fixing or upgrading the model if and when any security issues or other problems come up. And you can't use OpenAI, obviously.

Do you have a favorite infra for hosting models?