That just means the business automation is not deterministic, and highlights the issue with LLMs. The solution is not to switch providers but instead make the LLMs generate something that is deterministic given the same input.

> That just means the business automation is not deterministic, and highlights the issue with LLMs. The solution is not to switch providers but instead make the LLMs generate something that is deterministic given the same input.

I'm thinking of stuff that would typically be outsourced (like compliance stuff for identity verification et al), where it's really really hard to have deterministic rules.

LLMs are definitely a step up in dealing with fuzzy inputs, and enables you to mostly automate stuff that really wasn't feasible before.

But you do need really good evals and monitoring to make this work effectively.

Ah that would make sense then. I was thinking strictly in terms of business logic for handling input of data from an external source; sort of glorified data validation and then a decision based on the input data.

You'd be crazy to spend LLM money on that, but its probably happening all over the place.

> you'll need a really comprehensive set of evals and moniotoring to ensure stuff doesn't break if/when you change providers

Unless your self-hosting open weight models, you need this even if you don't change providers, since you have no control of what's on the other side of the API.