metawake

Great suggestion!! this is exactly the right methodology for establishing confidence intervals.

I've added this to the roadmap as `--bootstrap N`:

    ragtune simulate --queries queries.json --bootstrap 5
    
    # Output:
    # Recall@5:  0.664 ± 0.012 (n=5)
    # MRR:       0.533 ± 0.008 (n=5)

The implementation would sample N random subsets from the query set (or corpus), run each independently, and report mean ± std.

This also enables detecting real regressions vs noise eg "Recall dropped 3% ± 0.8%" is actionable, "dropped 3%" alone isn't.

Will ship this during next few weeks. Thanks for the push toward more rigorous methodology, this is exactly what's missing from most RAG benchmarks.

Author here. Built RagTune to stop guessing at RAG configs.

Surprising findings:

1. On legal text (CaseHOLD), 1024 chunks scored WORST (0.618). The "small" 256 chunks won (0.664). 7% swing.

2. On Wikipedia text? All chunk sizes hit ~99%. No difference.

3. Plot twist: At 5K docs, optimal chunk size FLIPPED from 256→1024. Scale changes everything.

Code is MIT: github.com/metawake/ragtune

Happy to discuss methodology.

Thanks! To answer your questions:

*Backends:* Currently supports Qdrant, pgvector, Weaviate, Chroma, and Pinecone. Adding more is straightforward since it's just implementing a Store interface. Let me know if I missed some good backend!

*Relevance scoring:* No LLM-as-judge — that's intentional. RagTune focuses on retrieval-layer metrics only:

- Vector similarity scores (what the DB returns) - Recall@K, MRR against your golden set - Score distribution diagnostics

The philosophy is: debug retrieval separately from generation. If your retrieval is broken, no amount of prompt engineering will fix it.

For chunk size/overlap optimization — exactly the use case! `ragtune compare --chunk-sizes 256,512,1024` lets you see the impact directly.

Happy to hear feedback if you try it!

I am using a vector DB using Docker image. And for debugging and benchmarking local RAG retrieval, I've been building a CLI tool that shows what's actually being retrieved:

  ragtune explain "your query" --collection prod

Shows scores, sources, and diagnostics. Helps catch when your chunking or embeddings are silently failing or you need numeric estimations to base your judgements on.

Open source: https://github.com/metawake/ragtune

I made a small project (https://github.com/metawake/puppetry-detector) to detect this type of LLM policy manipulation. It's an early idea using a set of regexp patterns (for speed) and a couple of phases of text analysis. I am curious if it's any useful, I created integration with Rebuff (loss security suite) just in case.

Hi HN,

I’ve been exploring whether prompt compression — done before sending input to LLMs — can help cut down on token usage and cost without losing key meaning.

Instead of using a neural model, I wrote a small open-source tool that uses handcrafted rules + spaCy NLP to reduce prompt verbosity while preserving named entities and domain terms. It’s mostly aimed at high-volume systems (e.g. support bots, moderation pipelines, embedding pipelines for vector DBs).

Tested it on 135 real prompts and got 22.4% average compression with high semantic fidelity.

GitHub: https://github.com/metawake/prompt_compressor

Would love feedback, use cases, or critiques!