Wow, that's a huge drop in quality. Perhaps it would make sense to use LLMs as a either a help for your customer support team, or as a fallback for when your customer support team does not have the bandwidth to help more customers at the time.

Also, can you go into more detail on the solution? E.g. how did you tailor it to your business (e.g. fine tuning vs no fine tuning, etc.)