News for hackers. Stuff that matters.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by aik on Friday December 01, 2023 @11:46PM (#38495962)
No method today is a panacea. RAG naturally isn’t the approach you would take if you’re trying to summarize a large body of text, as in your example. That doesn’t make sense. That is not what it is for.

What it’s awesome for is asking a somewhat specific question about some thing in a book if it can be answered by reviewing a paragraph or a few across the entire book.

So to answer your question: Start by using the right tool for the job. The problem with this is: Some tools are prohibitively expensive, challenging, or very time-consuming to use.

Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by deegles on Saturday December 02, 2023 @01:30AM (#38496455)
But the issue remains where your questions have to have easy "retrieval" aspects. You can't ask general questions that a human would have no trouble with. A book report is the example I used because you need to have read the entire book. essentially a book-length context window.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by namibj on Saturday December 02, 2023 @01:36PM (#38500795)
Thankfully, there are methods (like causal transformers, but also some question/text/answer architectures) for not re-computing self-attention between the document/knowledge-base tokens across the iterations of autoregressive decoding for generating the response.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by aik on Saturday December 16, 2023 @10:05AM (#38664716)
Do you have an example of where these methods still produce good summaries? Eg if you adjust how re-computation of self-attention in autoregressive decoding / between token generations works to significantly decrease the amount of computation needed?
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by bradknowles on Sunday December 03, 2023 @10:46PM (#38513516)
The real problem is knowing what tools exist for the job.

Or that the right tool actually does exist.

I would submit this is true for many fields of knowledge.