Also, can you go into more detail on the solution? E.g. how did you tailor it to your business (e.g. fine tuning vs no fine tuning, etc.)
I want to avoid having to talk to someone from a business at all costs. The only time I (begrudgingly) reach out to customer support is when I need to accomplish something that I can't do natively through the service. If a LLM can do the task, then just make it part of the service. Why add a LLM middleman?
I do get that I might be a bit weird and there are folks out there that like to talk to someone. But these people are probably even less likely to care for LLMs than I am, since they value talking to a real person.
We have done deep dives with a couple other providers who are being waaaay more insightful and smart about decomposing the problem.
These tools are not mature enough, no automated decision making process should leverage these tools without human intervention, especially when handling other humans. I'm not saying all LLM models are bad, I'm saying I'm talking about their architectural foundation, which is prone to these problems.
Our measurement was the rate of losing customers compared with them opening tickets complaining about some issue. We had roughly 12x more losses when using the LLM as a replacement for our support team in our A/B test.