News for hackers. Stuff that matters.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by deegles on Saturday December 02, 2023 @01:30AM (#38496455)
But the issue remains where your questions have to have easy "retrieval" aspects. You can't ask general questions that a human would have no trouble with. A book report is the example I used because you need to have read the entire book. essentially a book-length context window.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by namibj on Saturday December 02, 2023 @01:36PM (#38500795)
Thankfully, there are methods (like causal transformers, but also some question/text/answer architectures) for not re-computing self-attention between the document/knowledge-base tokens across the iterations of autoregressive decoding for generating the response.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by aik on Saturday December 16, 2023 @10:05AM (#38664716)
Do you have an example of where these methods still produce good summaries? Eg if you adjust how re-computation of self-attention in autoregressive decoding / between token generations works to significantly decrease the amount of computation needed?