Solving the out-of-context chunk problem for RAG

Re:Solving the out-of-context chunk problem for RAG by simonw on Thursday July 25, 2024 @02:24AM (#41065315)

It turns out picking that threshold is extremely difficult - I've tried! The value seems to differ for different searches, so picking eg 0.7 as a fixed value isn't actually as useful as you would expect.

[ Parent | on HN ]

Re:Solving the out-of-context chunk problem for RAG by zmccormick7 on Thursday July 25, 2024 @11:08AM (#41069576)

Agreed that thresholds don't work when applied to the cosine similarity of embeddings. But I have found that the similarity score returned by high-quality rerankers, especially Cohere, are consistent and meaningful enough that using a threshold works well there.

[ Parent | on HN ]

Re:Solving the out-of-context chunk problem for RAG by kgeist on Monday July 29, 2024 @06:04AM (#41099279)

I use similarity threshold (to remove absolutely irrelevant results) and then use a reranker to get Top N.

[ Parent | on HN ]