Re:Show HN: Paramount, an OSS Package for *Human* Evals of A...
by bee_rider on Friday June 14, 2024 @11:37AM
(#40681829)
It seems like a very convoluted path, but if people are for some strange reason more apt to end up at the FAQ via “chatting” with a tool like this rather than just looking it up directly, this still seems useful, right? I mean what’s the alternative, is there a “more self-directed users as a services” offering out there?
Re:Show HN: Paramount, an OSS Package for *Human* Evals of A...
by hakimk on Saturday June 15, 2024 @10:36AM
(#40690075)
FAQ/knowledgebase are not enough to reach 95%. If that was the case then this problem would've been solved long ago with the advent of semantic search engines.
You also need the following:
1) General principles of instructions split across topics - essentially distilling a support training manual into a flow chart of prompts.
2) Algorithms to detect best past resolutions to create new knowledge-base material on the go.
3) Continuous retrain and feedback loops which we have written models to perform on behalf of customers.
And continuous R&D effort into how these systems can improve.
Is this the 95% of issues that are common between customers that could be and are handled at scale with a FAQ/knowledgebase that is surfaced to a user automatically through prompts?
How does it actually do on the smaller percentage of questions that are complex?
And that 95% is on the most successful projects. But what does the distribution of accuracy look like across all projects? If 2 projects out of 1000 are 95% accuracy that doesn't mean much to me.