Some things that have were notable:
Model pipelines tend to be flakier than other pipelines you have. They are much more complicated, and it can be easy to hit a resource limit if you aren't careful, or have a unhandled exception accidentally kill a pipeline 10hrs into it.
Avoiding those outright is obviously the best path, but that can be easier said than done.
One thing that we found really helpful was creating an error record in a database for every piece of data that failed to get processed, where it failed in the pipeline, etc. Retries, and alters were easy to tack on after that.
Could you share some names for someone inexperienced in MLOps to do some research?
Kubeflow is the one I see used the most (self hosted). The public clouds also have products targeting these workloads.
Ad tech
How did you develop and launch it?
New features for an existing product. Most of them were switched on globally when we launched them.
Which LLM(s) are you using (e.g. GPT-4, Claude, open source models)?
We started using language models before ChatGPT and friends came out. We built our models from scratch, and still use and update them.
What's your revenue model?
Saas
How much are you making (if you want to share, ballpark figures are fine)?
We don't charge extra for features that use ML, it's an implementation detail that our users don't care about. So, I am not sure what % of the whole they contribute to. Its not the majority.
What challenges have you faced?
"mlops" was a real challenge in the early days, and we built so many bespoke distributed systems to figure it out. Eventually we did, but finding the right balance between building a performant system and a system that doesn't explode in cost was tough. There is off the shelf things that do this these days, but they can be expensive.
Any advice for others looking to enter this space?
Solve problems customers are willing to pay for. Some of the features we have dreamed up have been total flops. Many of the features we have developed (ml or otherwise), while the add value for the customer and add to the moat, they aren't valuable enough that our users are willing to pay extra for it as a standalone product.