RAG accuracy jumped from 10% to 60% when I added outcome scoring

Re:RAG accuracy jumped from 10% to 60% when I added outcome ... by ramenlover on Sunday December 21, 2025 @11:23AM (#46345923)

How did you measure the 60% improvement rate?

[ Parent | on HN ]

Re:RAG accuracy jumped from 10% to 60% when I added outcome ... by roampal on Sunday December 21, 2025 @11:37AM (#46346041)

Ran a 4-way comparison test across 200 query-memory pairs:

- Baseline RAG (embedding similarity only): 10%

- RAG + reranker: 20%

- Outcomes only (no reranker): 60%

- RAG + outcome scoring (mature memories with 20+ uses): 60%

"Accuracy" = correct memory ranked #1 for the query. The outcome scoring uses Wilson score lower bound - memories that consistently get positive feedback from the "user" get boosted, ones that fail get demoted.

Test methodology: https://github.com/roampal-ai/roampal/blob/main/dev/benchmar...

[ Parent | on HN ]