News for hackers. Stuff that matters.
Re:Ask HN: Is RAG the Future of LLMs? by spencerchubb on Sunday April 14, 2024 @06:44PM (#40035161)
I believe RAG is a temporary hack until we figure out virtually infinite context.

I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on.

RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimize loss, it is just a similarity score.

Obligatory note that I could be wrong and it's just my armchair opinion

Re:Ask HN: Is RAG the Future of LLMs? by simonw on Sunday April 14, 2024 @07:08PM (#40035320)
It's not just about context length, it's about performance.

A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.

As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

Re:Ask HN: Is RAG the Future of LLMs? by verdverm on Sunday April 14, 2024 @07:21PM (#40035408)
There's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context.

Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions

Re:Ask HN: Is RAG the Future of LLMs? by spencerchubb on Sunday April 14, 2024 @07:27PM (#40035447)
Good point! If datasets get that large that is an argument in favor of RAG
Re:Ask HN: Is RAG the Future of LLMs? by 2099miles on Sunday April 14, 2024 @07:55PM (#40035620)
As a concept infinite context will not be discovered. We can spin up huge models with large contexts that cost more but even 1M context is 1 expensive and 2 not 3trillion(size of some datasets)

Rag is not bad attention, rag is the user not knowing the context to give the LLM

Re:Ask HN: Is RAG the Future of LLMs? by ttul on Sunday April 14, 2024 @07:16PM (#40035383)
https://arxiv.org/html/2404.07143v1

“This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”

Re:Ask HN: Is RAG the Future of LLMs? by mikecaulley on Sunday April 14, 2024 @06:47PM (#40035188)
This doesn’t consider compute cost; the RAG model is much more efficient compared to infinite context length.
Re:Ask HN: Is RAG the Future of LLMs? by epistasis on Sunday April 14, 2024 @07:00PM (#40035257)
Agreed. I think that RAG implemented via tool-calling, with multiple agents talking to each other, is a much much more likely evolution in the future versus a single unified model.

I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.

Re:Ask HN: Is RAG the Future of LLMs? by darby_eight on Sunday April 14, 2024 @07:12PM (#40035350)
> I believe RAG is a temporary hack until we figure out virtually infinite context.

I'd assume "large enough" context is the actual goal here, not "virtually infinite".

Re:Ask HN: Is RAG the Future of LLMs? by choilive on Sunday April 14, 2024 @06:58PM (#40035247)
I think that "next level" would essentially be a RAG referential information system that gets data via search engines or databases. Maybe we will have the "google search" equivalent completely intended for LLM clients where all data is stored, searched, and returned via vector embeddings, but it could tap on exabytes of information.
Re:Ask HN: Is RAG the Future of LLMs? by 0x008 on Monday April 15, 2024 @05:00AM (#40038241)
That is actually already what is happening, tools like llama_index are trying to provide all the necessary integrations.
Re:Ask HN: Is RAG the Future of LLMs? by fallingknife on Sunday April 14, 2024 @07:15PM (#40035377)
That's going to be quite a while then because LLMs don't even perform well at anything close to the current context limit. They will miss important details when the context gets over a couple thousand tokens.