hakimk

FAQ/knowledgebase are not enough to reach 95%. If that was the case then this problem would've been solved long ago with the advent of semantic search engines.

You also need the following:

1) General principles of instructions split across topics - essentially distilling a support training manual into a flow chart of prompts.

2) Algorithms to detect best past resolutions to create new knowledge-base material on the go.

3) Continuous retrain and feedback loops which we have written models to perform on behalf of customers.

And continuous R&D effort into how these systems can improve.

We typically offer a package price for companies with >1k questions per month with white-glove onboarding and numerous custom optimizations to ensure higher performance.

For the fully self-serve pay as you go option, we are able to offer a lower pricing.

We have a number of safeguards:

1) Our customers can decide which categories of end-user questions should be auto-escalated

2) We ensure that if a question cannot be answered based directly on knowledgebase content, we either escalate or try to ask followups to drill deeper

3) We offer an API which our customers can directly integrate with their own systems. For protection, we either limit the set of allowed actions, or if the customer prefers, they can handle the safeguarding code internally with advice from us.

Lastly I would say the usage of LLMs is young in the support space and things will only improve from here!

Indeed the evaluations are saved to DB. Right now it's possible to use this for regression testing with the help of the Optimize tab. In the optimize tab you can experiment with changing an input parameter (such as prompt or temperature, etc), then rerun recordings and see whether the the LLM response matches all previously accepted recordings or not - to see a similarity score which tells you if your change introduced regressions or not.

In the future we are planning to enable a retraining pipeline - most likely we will do this in our core offering at usefini.com

I understand where you're coming from - poor support experiences are are the bane of my existence.

Preventing poor support experiences is the exact reason we started Fini. With a long career as a software engineer in the support industry, I have seen first hand when and why these initiatives fail.

Unfortunately, when growth-stage companies experience a surge of signups, they face the choice between either 1) delay responding to all support issues by weeks, due to lack of staffing, or 2) trying to automate/offshore support, leading to poor coverage and accuracy.

Both of these options usually lead to horrendous support experiences such as the ones you mention.

Thankfully, with recent advances such as LLMs, we are one step closer to bridging this gap. In our most successful projects we are able to reach 95% accuracy specifically thanks to keeping a close eye on human evaluations.

That's why we are launching this package as well, which we believe brings us one step closer towards our mission of helping millions of end-users receive the high quality support they deserve within seconds rather than weeks.

It's a challenge which very few (if any!) companies have pulled off well, which is why we have decided to put all our effort into making scalable and hassle-free support our first priority.

Thanks for the feedback everyone, we have now removed this paragraph and the whole package is now under GPL regardless of usage!

The idea was to get in touch with enterprises looking to make heavy use of the project - but you're right that this may have unintended effects.

Hey HN, Hakim here from Fini (YC S22). We've seen first hand how AI chat projects pan out, and so have released an OSS library to ensure the industry gets more tools for improving outcomes.

Many AI chat projects are scrapped due to persistent inaccuracies in LLM responses. Paramount is an open-source Python package designed to bridge the gap between LLM-generated and ideal responses by incorporating expert feedback directly into the evaluation process. It provides a robust framework for recording LLM function outputs (ground truth data) and facilitates agent evaluations, reducing the time to identify and correct errors.

Developers can integrate Paramount with a decorator that logs LLM interactions into a CSV or database, followed by a straightforward UI for expert review. This process accelerates the debugging and validation phase of your project and de-risks your launch.