Faster Retrieval for Better, More Efficient RAG

Re:Faster Retrieval for Better, More Efficient RAG by armcat on Wednesday November 11, 2026 @02:34PM (#50457462)

Just yesterday I saw a talk from Spotify where they presented their own fast retrieval mechanism "Parallel Fusion Router" [1]. It's also in the "learned retrieval" paradigm - in their case they use a teacher LLM to train a much smaller router LLM how to retrieve, using an LLM-as-a-judge based reward function.

From my own experience, these learned methods from Apple, Spotify and others, are highly relevant to hyperscale only - when you are dealing with hundreds of millions of items to choose from (or more), and require super high relevance in the top-5. For all other use cases (up to a few million items) I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods (though the latter are more publishable in recsys and icml).

[1] https://research.atspotify.com/2025/9/you-say-search-i-say-r...

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by bob1029 on Wednesday November 11, 2026 @07:03PM (#50460893)

[delayed]

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by fcarraldo on Wednesday November 11, 2026 @08:42PM (#50461921)

> I found that just building a BM25 index, and giving the LLM a BM25 "search tool" and casting a "wide net" wins every single time compared to learned methods

Funny, I did exactly this for my last RAG project. Works great for a small dataset.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by greesil on Wednesday November 11, 2026 @11:13PM (#50462997)

For some reason I thought everyone was using BERT vectors and cosine similarity to do this RAG stuff. This is not my jam, but I do dabble. Neat!

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by freakynit on Wednesday November 11, 2026 @11:24PM (#50463061)

Same here. Up to a few millions of documents, this is already great. Combine this with a few other simple techniques and extra metadata during ingestion, and you already have a system that practically useable.

I have tried this by ingesting all of cloudflare's highly technical complex documents (more than 3000 A4 sized pages) and then comparing results with that of cloudflare's own chatbot in their dashboard.

In all cases my system either matched, or performed marginally better than cloudflare's own bot.

I do use a few more techniques though, but, not RDF triplets, and definitely no custom trained neural network.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by totetsu on Wednesday November 11, 2026 @11:46PM (#50463163)

It seems, that the amount of information that "typically" an person would want to work with, can be handled with pretty much any technique by the compute that most of us have, at a performance level that is satisfying.. It's only when you start to increase the information volume, decrease the compute, or increase the performance expectations, that it is really impactful to start to find methods that really make the most of the deep structures of that information to query it.

[ Parent | on HN ]