Re:What Is Retrieval-Augmented Generation a.k.a. RAG?
by filterfiber on Friday December 01, 2023 @04:45PM
(#38492905)
I have to agree, RAG is effectively just compensating for data not fitting in the context window, and it has (potentially severe) limitations.
Summarizing a book I think is a great example, RAG prevents LLMs from correlating all of the data.
It will likely never be obsolete, but a larger context would be far preferable in nearly all scenerios.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG?
by svcrunch on Friday December 01, 2023 @05:43PM
(#38493576)
I think your comment is accurate, but regarding your last point:
"However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated."
How does one solve the problem of access-controlled data, if not through RAG? Do you imagine a separate version of the LLM for every user, reflecting their unique permissions on the data?
Also, in scenarios where the data is being updated regularly, RAG provides much lower latency to the new information. Deletes also present a challenge for a pure-LLM approach.
Other modes of RAG that retrieve multiple times per generation (e.g. every so-many tokens) can improve performance, and intuitively feel closer to the 'sparse attention over the entire corpus' that RAG is approximating.
However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated.