Faster Retrieval for Better, More Efficient RAG

Re:Faster Retrieval for Better, More Efficient RAG by goldenarm on Wednesday November 11, 2026 @01:32PM (#50456793)

Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by dv_dt on Wednesday November 11, 2026 @01:40PM (#50456881)

If the data is this large then why not ask the llm to build a query

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by aaroninsf on Wednesday November 11, 2026 @01:45PM (#50456938)

My case:

With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by nostrebored on Wednesday November 11, 2026 @01:46PM (#50456947)

latency is often the reason, distilling interactions into significantly better retrieval is the second.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by itishappy on Wednesday November 11, 2026 @02:12PM (#50457225)

That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by ozkatz on Wednesday November 11, 2026 @02:36PM (#50457489)

all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by jdthedisciple on Wednesday November 11, 2026 @02:51PM (#50457722)

> red|maroon|scarlet|crimson

still doesn't include typo'ed variants for any of these

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by ozkatz on Wednesday November 11, 2026 @04:17PM (#50458801)

but a good combination of prompt, harness and model will easily close that gap. a more advanced (but equivalent) form of grep could also offer fuzzy matching/levenshtein distance function to augment that.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by c-hendricks on Wednesday November 11, 2026 @04:56PM (#50459239)

This sounds like AOT vs JIT weights.

[ Parent | on HN ]

Re:Faster Retrieval for Better, More Efficient RAG by everforward on Wednesday November 11, 2026 @08:36PM (#50461869)

It will, but it does so by burning tokens. I'm sure you've also seen the output of an overly eager grep by an LLM.

I think we're verging on the point where people will care more about the most token-efficient ways to do a task than the lowest human effort version. Humans are expensive, humans using half as many tokens are cheaper.

[ Parent | on HN ]