Embeddings just aren’t the most interesting thing here if you’re running a frontier fm.

Besides, if you could reliably verify results, you’ve essentially built an RL harness—which is a lot harder to do than building an effective search system and probably worth more.

0: https://github.com/dmitrym0/dm-gptel-simple-org-memory

Anthropic found embeddings + BM25 (keyword search) gave the best results. (Well, after contextual summarization, and fusion, and reranking, and shoving the whole thing into an LLM...)

But sadly they didn't say how BM25 did on its own, which is the really interesting part to me.

In my own (small scale) tests with embeddings, I found that I'd be looking right at the page that contained the literal words in my query and embeddings would fail to find it... Ctrl+F wins again!

I think it's just a case of "this isn't something we need to solve, other companies solve it already and then our thing can integrate with that."

Or maybe it's really just marginal gains compared with iterative grepping. I don't know. (Still amazed how well that works!)

https://cursor.com/docs/context/codebase-indexing

Seems like sometimes Cursor has a better understanding of the vibe of my codebase than Claude code, maybe this is part of it. Or maybe it’s just really marginally important in codebase indexing. Vector dbs still have a huge benefit in less verifiable domains.

For most cases though sticking with BM25 is likely to be "good enough" and a whole lot cheaper to build and run.

http://search-sensei.s3-website-us-east-1.amazonaws.com/

(warning! It will download ~50MB of data for the model weights and onnx runtime on first load, but should otherwise run smoothly even on a phone)

It runs a small embedding model in the browser and returns search results in "real time".

It has a few illustrative examples where semantic search returns the intended results. For example bm25 does not understand that "j lo" or "jlo" refer to Jennifer Lopez. Similarly embedding based methods can better deal with things like typos.

EDIT: search is performed over 1000 news articles randomly sampled from 2016 to 2024

In general RAG != Vector Search though. If a SQL query, grep, full text search or other does the job then by all means. But for relevance-based search, vector search shines.

Vector DB's and embeddings are dead simple to figure out, implement, and maintain. Especially for a local RAG, which is the primary context here. If I want to find my latest tabular notes on some obscure game dealing with medical concepts, I should be able to just literally type that. It shouldn't require me remembering the medical terms, or having some local (or god forbid, remote) LLM iterate through a dozen combos.

FWIW I also think this is a matter of how well one structures their personal KB. If you follow strict metadata/structure and have coherent/logical writing, you'll have better chance of getting results with text matching. For someone optimizing for vector space search, and minimizing the need for upfront logical structuring, it will not work out well.

Claude Code is widely regarded to be the best coding agent tool right now and it uses search, not embeddings.

I use it to answer questions about files on my computer all the time.

- Iterate over your docs to build eval data: hundreds of pairs of [synthetic query, correct answer]. Focus on content from the docs not general LLM knowledge.

- Kick off a few parallel evaluations of different RAG configurations to see what works best for your use case: BM25, Vector, Hybrid. You can do a second pass to tune parameters: embedding model, top k, re-ranking, etc.

I build a free system that does all this (synthetic data from docs, evals, test various RAG configs without coding each version). https://docs.kiln.tech/docs/evaluations/evaluate-rag-accurac...

But if you want to build truly great search that's the approach to take.

You're realistically going to need chunks of some kind anyway to feed the LLM, and once you got those it's just a few lines of code to get a basic persistant ChromaDB going.