That's excellent advice, the only downside being that collecting that eval data remains difficult and time-consuming.
But if you want to build truly great search that's the approach to take.
Agree totally. I’m spending half my time focused that problem (mostly synthetic data gen with guidance), and the other half on how to optimize once it works.
At this point you could also optimize your agentic flow directly in DSPy using a colbert model / Ratatouille for retrieval.
- Iterate over your docs to build eval data: hundreds of pairs of [synthetic query, correct answer]. Focus on content from the docs not general LLM knowledge.
- Kick off a few parallel evaluations of different RAG configurations to see what works best for your use case: BM25, Vector, Hybrid. You can do a second pass to tune parameters: embedding model, top k, re-ranking, etc.
I build a free system that does all this (synthetic data from docs, evals, test various RAG configs without coding each version). https://docs.kiln.tech/docs/evaluations/evaluate-rag-accurac...