Re:Solving the out-of-context chunk problem for RAG
by kgeist on Thursday July 25, 2024 @01:41AM
(#41065077)
>but because embeddings search orders by similarity score it will ALWAYS return results, really scraping the bottom of the barrel if it has to
Why not have a similarity threshold? Say, if the distance is below 0.7, do not accept the search result.
Re:Solving the out-of-context chunk problem for RAG
by simonw on Thursday July 25, 2024 @02:24AM
(#41065315)
It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.
Re:Solving the out-of-context chunk problem for RAG
by zmccormick7 on Thursday July 25, 2024 @11:08AM
(#41069576)
Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.
Re:Solving the out-of-context chunk problem for RAG
by kgeist on Monday July 29, 2024 @06:04AM
(#41099279)
I use similarity threshold (to remove absolutely irrelevant results) and then use a reranker to get Top N.
Re:Solving the out-of-context chunk problem for RAG
by jairuhme on Thursday July 25, 2024 @11:11AM
(#41069610)
I'll add to what the other commenter noted, but sometimes the difference between results get very granular (i.e. .65789 vs .65788) so deciding on where that threshold should be is little trickier.
The other problem is that embeddings search can miss things that a direct keyword match would have caught. If you have key terms that are specific to your corpus - product names for example - there's a risk that a vector match might not score those as highly as BM25 would have so you may miss the most relevant documents.
Finally, embeddings are much more black box and hard to debug and reason about. We have decades of experience tweaking and debugging and improving BM25-style FTS search - the whole field of "Information Retrieval". Throwing that all away in favour of weird new embedding vectors is suboptimal.