Grep is the fastest tool for many things, but for production RAG it cannot search with meaning, different languages or typos like hybrid vector search can. And it won't scale on terabytes of documents across distributed servers if you want to say under 1000ms.

If the data is this large then why not ask the llm to build a query

My case:

With a very large dataset of totally unnormalized data, semantic folding is a silver bullet.

latency is often the reason, distilling interactions into significantly better retrieval is the second.

That's more-or-less what RAG is; a tool to surface relevant data so that AI can make informed decisions. Efficient retrieval is important so that AI doesn't need to ingest the whole corpus for each query.

all true for a single grep invocation, however a capable LLM can drive grep using similar terms (`grep (red|maroon|scarlet|crimson)`, moving the measure of similarity from embedding/indexing time to inference time. There's benefits to that.

> red|maroon|scarlet|crimson

still doesn't include typo'ed variants for any of these

but a good combination of prompt, harness and model will easily close that gap. a more advanced (but equivalent) form of grep could also offer fuzzy matching/levenshtein distance function to augment that.

This sounds like AOT vs JIT weights.

It will, but it does so by burning tokens. I'm sure you've also seen the output of an overly eager grep by an LLM.

I think we're verging on the point where people will care more about the most token-efficient ways to do a task than the lowest human effort version. Humans are expensive, humans using half as many tokens are cheaper.

Grep is not semantic like embeddings, although the llm can expand a concept into multiple literal searches with a similar effect.

Your understanding was applied too strictly and you missed the intent of the joke...

What's the joke?

There have been some attempts at "semantic grep" but don't know its current status

https://github.com/lightonai/next-plaid/tree/main/colgrep

Considering the exemple they have provided, it seems they used grep...

Retrieval is a whole subject of study. For well-structured plaintext datasets, basic filtering may be as good as semantic and lexical searches. Takes measurement to know what you need

the amount of times the clanker has used grep badly and nuked my changes should be illegal, god bless git, god bless AI hook gates, and god bless RAG

You definitely should use grep wherever you can (i.e. simple, unranked term searches) to 1) exclude things you are definitely not care about 2) to include things you definitely do care about. The reason is that this is cheap and easy and it reduces the overall results you still need to dig through. And because everything else is much more expensive this is going to help reduce the amount of compute needed from that. And results are cacheable as well which helps you with repeat searches for similar things.

For ranked searches, you can do more than one and combine their results. Reciprocal rank fusion (RRF) is a common approach to do that. A simple ranked lexical query and one or more vector searches. Can be combined to amplify results that perform well across multiple of these.

RRF is really easy: for each search result, take its position in each ranked list, add 60, and divide 1 by that number. Add up those values across all lists to get the final score. Results that rank highly in multiple lists get higher scores. The magical number of 60 dampens the effect of highly ranked results biasing the outcome to much. It's not going to be a good approach for all data sets. But it works surprisingly well and it's stupidly easy to implement.

For term search, expending some AI and other cleverness at indexing time to e.g. calculate categorizations is very helpful. If you can reduce your result set with some simple term searches, it might be small enough to pass to an LLM directly and skip ranked searches entirely. Using vector search as magic pixie dust is generally going to work a lot better if you can filter and combine it with a lexical search. Using e.g. a decision model at index time to calculate some categorizations and classifications is relatively cheap and easy relative to calculating vector embeddings.