Show HN: RagTune – EXPLAIN ANALYZE for your RAG retrieval layer
https://github.com/metawake/ragtune
CLI tool to debug and benchmark RAG retrieval without LLM calls.- `ragtune explain "query"` → see what was retrieved with scores - `ragtune simulate` → batch eval with recall/MRR metrics - `ragtune compare` → compare embedders or chunk sizes - CI/CD mode for quality gates
Works with Qdrant, pgvector, Weaviate, Chroma, Pinecone.
Built because I kept guessing why retrieval was bad. Now I can see exactly what's happening.
(
2 comments
| on HN
)
Quick question: Does this work with any vector database (Pinecone, Weaviate, etc.) or is it specific to certain backends?
Also curious how you're measuring relevance - are you using LLM-as-judge or some other scoring method?
This could be really useful for optimizing chunk size and overlap settings!
*Backends:* Currently supports Qdrant, pgvector, Weaviate, Chroma, and Pinecone. Adding more is straightforward since it's just implementing a Store interface. Let me know if I missed some good backend!
*Relevance scoring:* No LLM-as-judge — that's intentional. RagTune focuses on retrieval-layer metrics only:
- Vector similarity scores (what the DB returns) - Recall@K, MRR against your golden set - Score distribution diagnostics
The philosophy is: debug retrieval separately from generation. If your retrieval is broken, no amount of prompt engineering will fix it.
For chunk size/overlap optimization — exactly the use case! `ragtune compare --chunk-sizes 256,512,1024` lets you see the impact directly.
Happy to hear feedback if you try it!