News for hackers. Stuff that matters.
Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by kristjansson on Friday December 01, 2023 @03:47PM (#38492145)
This doesn't deserve the downvotes it (currently) has. This is a completely accurate statement, simple RAG works well when questions are highly correlated with specifics chunks of documents. It does not allow an LLM to synthesize an entire corpus to an answer (e.g. a book report).

Other modes of RAG that retrieve multiple times per generation (e.g. every so-many tokens) can improve performance, and intuitively feel closer to the 'sparse attention over the entire corpus' that RAG is approximating.

However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated.

Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by filterfiber on Friday December 01, 2023 @04:45PM (#38492905)
I have to agree, RAG is effectively just compensating for data not fitting in the context window, and it has (potentially severe) limitations.

Summarizing a book I think is a great example, RAG prevents LLMs from correlating all of the data.

It will likely never be obsolete, but a larger context would be far preferable in nearly all scenerios.

Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by svcrunch on Friday December 01, 2023 @05:43PM (#38493576)
I think your comment is accurate, but regarding your last point:

"However, fine-tuning on relevant, high quality, knowledge-rich question/answer pairs seems dominant, when such examples are available or can be generated."

How does one solve the problem of access-controlled data, if not through RAG? Do you imagine a separate version of the LLM for every user, reflecting their unique permissions on the data?

Also, in scenarios where the data is being updated regularly, RAG provides much lower latency to the new information. Deletes also present a challenge for a pure-LLM approach.

Re:What Is Retrieval-Augmented Generation a.k.a. RAG? by willy_k on Friday December 01, 2023 @08:19PM (#38494831)
I don’t know much about the space, but couldn’t LoRA be used to (moderately) efficiently use a relevant and individual fine-tuned model alongside a general LLM?