Re:Faster Retrieval for Better, More Efficient RAG
by jillesvangurp on Thursday November 12, 2026 @03:08AM
(#50464365)
You definitely should use grep wherever you can (i.e. simple, unranked term searches) to 1) exclude things you are definitely not care about 2) to include things you definitely do care about. The reason is that this is cheap and easy and it reduces the overall results you still need to dig through. And because everything else is much more expensive this is going to help reduce the amount of compute needed from that. And results are cacheable as well which helps you with repeat searches for similar things.
For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.
RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.
For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.
For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.
RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.
For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.