We taught a small LLM to throw away 68% of our RAG context

Re:We taught a small LLM to throw away 68% of our RAG context by rooftopzen on Monday July 06, 2026 @06:30PM (#48811342)

Cliche topic - from a few years ago (the "RAG is dead" vs "All You Need Is Advanced RAG" BS - it came in waves and cycles, spread by bots on social media networks).

"Pruning RAG Context" is trying to recycle the old stuff (again), presuming the reader is naive (implies kapa.ai is not going anywhere). The current cycles were "openclaw" (I think that died), now we are on "harnesses" - when that dies the paid social media bots will give you something else. Shell game.

Just declare / define dictionary as a variable in your prompt to carry forward (when you decide to continue using LLMs for certain things). Also either summarize or truncate history. 3-4 year old concept. Not a big thing.

[ Parent | on HN ]

Re:We taught a small LLM to throw away 68% of our RAG context by timcobb on Monday July 06, 2026 @07:21PM (#48811778)

Shallow dismissal

[ Parent | on HN ]

Re:We taught a small LLM to throw away 68% of our RAG context by wolvoleo on Tuesday July 07, 2026 @02:19AM (#48814267)

"RAG is dead" was declared because of ever expanding context windows.

However, resources consumed expand so much with context that I think there is a huge practical barrier there. In the beginning AI services were basically free so nobody cared but this is rapidly shrinking.

Personally I see more in a combination of RAG and live querying during the thinking process (e.g. by tools).

Also I don't think dumping any context that might be relevant into the model really helps accuracy. In my experience models just get lost when they get an overload of irrelevant stuff in their context and start overlooking the relevant parts even if it does fit the window.

[ Parent | on HN ]