(Obviously this doesn’t apply to searching actual rich document data - for that, go all in on text search, embedding, etc)
I want fuzzy search like 95% of time and then I might consider having additional list of things that can be suggested by vector search.
Volume of documents, size of documents, versioning, frequency of update, documents similar or overlapping information, how much or exactly what you need for the LLM to understand, AI friendly documents, who has access and at what level, blue teaming, red teaming, multi-lingual, does the LLM know the domain language of the user and documents.
I probably missed a few things even with that.
I think there needs to be a linguist version of "what every programmer needs to know about (full?) text search"...
I'm not a linguist and I don't study languages, but I know enough to realize if a text search system is not designed for a particular language, it simply won't work. (As an example, to implement English search in a system for a hobby project, I had to import a US/UK spelling wordlist, and implement the Porter Stemming Algorithm. This is just for "one" language, and probably does not cover the other "English" dialects. Imagine doing a different workaround for every language in existence...)
RAG is actually a very language-agnostic way to work around those issues.
It was one of the most fun projects I've worked on in my career so far. I got a learn a lot about how US and international addresses worked, so many edge cases, and got to really understand how customers were using the existing search to make sure they weren't adding any duplicates to the database. Token filters and synonyms were neat and figuring out the right indexing strategy was a lot of fun.
It was a lot more work to get it right for most of the use-cases our customers had than just "throw it into ES and be done". That would probably have been fine for the 80/20 case, like you said, but I agree that the bulk of the work is going to be fine-tuning the search solution, whatever technology you're using.
We were pretty invested in using Google at the time for their Geocoding API. The questions that needed answering for our use-case were:
- Does this place actually exist?
- Can it be found on Google Maps?
- Do we (the company/researchers) have an entry for this place already?
It is not psychological, it is fully justified: substring search cannot find synonyms, periphrases and mistaken neighbours.
On the other hand, your post may contain a good idea: L=instruct_LLM("provide a list of synonyms and periphrases of terms T within context C", T, C); then iter(`grep l in L`). One NN query and a `grep` collection. But again, if one wanted to order the results, it is either through a dumb crierion or through another LLM query - but this could make it extremely costly (requiring either a huge context or a quadratic number of ordering queries).
And, the above `grep` based procedure would remain keyword based and not semantic based, which means that the user must know that it will not be based on comprehension but on the possible results that keyword matching can yield.
(BTW: you made me realize - I had to take "time off" for over half a year... I am sure I missed a lot.)
--
Edit: for clarity: for "full text search" we remain on the interpretation of "searching for literal substrings" - and whether plain user provided keywords list or LLM enriched list based on the former, and whether more or less successful, it remains a syntactic search quite distinct from a semantic one. Having an LLM enrich the original keywords list can be a good idea, but the possibility of misses remains when compared to a properly working semantic search.
You can see it in action if you watch the “chain of thought” text when using coding agents.
It is, if people don't even stop to think if they need synonyms, periphrases, or mistaken neighbours.
As the blog post points out, more often than not you don't, particularly if your primary usecase is to search for technical keywords or codenames.
Vectors and LLMs are great, but there's no magic pill here. If your parts data and config management[1] is all crazy, that's an institutional problem. Buying a crapton of tokens isn't fixing it, unless you're using it to help build an actual formal solution based on good fundamentals.
[1] Such as it is.
i dont believe ppl are building rag for this
What do you actually think people do when using LLMs to build AI coding agents?
You underestimate the ability of people to overengineer things.
https://www.anthropic.com/engineering/contextual-retrieval
This is from two years ago, but I think it's still SotA?
RAG is routing and decision making.
I found so much joy in achieving the best results given the requirements than simply hoping for the best with the cool kid called vector db and embeddings.
I agree with you.
Depending on the context and required output I decide how to orchestrate a multitude of specialized modules that produce the best specific result to gain a universally usable system.
It maintains itself.
Also live updates need reruns and rebuilding certain indexes. Everything is highly dynamic but in a deterministic way.
I found my niche with RAG selling and I build them myself.
I take pride in them.
So many look at the technology but not on the required output. It takes hours of talking to people to get an idea of what they need.
And there are regulated businesses where certain information is required to be always factual correct - pricing for example.
Vector search becomes a liability for this use case.
So naturally you have to reconsider your system: mixing factual with probabilistic content and how to make sure, it hits always certain quality benchmarks and on the other hand doesn’t fail others.
I love this kind of stuff.
And there is personal information etc.
Using modules is the key. Orchestration is really fun but I have to admit, not for the faint of heart.
And ever changing parts: LLMs, or restrictions to be matched liked autonomously working - I love RAG.
It gave me back the joy of developing. In fact I never had so much phun before, because it is also “team work”: I am not programming, I am managing a product.
I was in Senior Management of a top tier international bank and besides that build the only ever working platform or IT transformation called dbCORE and overlooked 13 teams with 120 developers.
RAG gives me dbCORE vibes so to say.
Good luck and fun with your RAG systems.
The trick was just to let the LLM come up with its own SQL queries for searching... and the results are impressive.
Before a lot of frameworks existed, you'd see DEVs taking user input on a web form, and then just throwing it directly at the MTA. So spammers could submit email@address\nCC: persontospam@address, and the like.
Now LLMs are a different beast, but you have input validation for LLMs, unique to all other validation methods. Yet there's actually no safe way to ever validate user input for a LLM, except for very rigid input validation on single words. Take the email example above. You'd need a regex to only validate an email address (and that isn't simple), but once you expand it to actually allowing sentences?
The LLM is now input validation vulnerable.
And that means no user input can be used in unvalidated commands.
And then just random hallucinations. I'm curious how the gp managed weirdo LLM behaviour, like out of the blue 'drop table' or accidental select into as opposed to just select.
>On this benchmark, a pure LLM generated an accuracy score of zero. Adding RAG, prompt engineering, and agentic AI raised accuracy to the 10+% range.
[1] Any text-to-SQL benchmark should address difficulties of real-world data stores (acm.org) (21 comments):
https://news.ycombinator.com/item?id=49013995
[2] If You Think You Can Do Real-World Text-to-SQL:
https://cacm.acm.org/blogcacm/if-you-think-you-can-do-real-w...
[3] BEAVER: An Enterprise Benchmark for Text-to-SQL:
Double hard if you're dealing with private market finance customers with their own ideas on what "common" terms mean. I tried to replace the embedding/ingestion pipeline multiple times and nothing I tried was better over a large amount of documents. Performance sucked, the agent was re-writing and re-trying queries over and over until it found what it wanted, and vector search with a little work up front was worlds better (though it was expensive)