News for hackers. Stuff that matters.
Re:Ask HN: Is RAG the Future of LLMs? by simonw on Sunday April 14, 2024 @07:08PM (#40035320)
It's not just about context length, it's about performance.

A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.

As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.

Re:Ask HN: Is RAG the Future of LLMs? by verdverm on Sunday April 14, 2024 @07:21PM (#40035408)
There's also the focus. A long context means the LLM is more easily distracted. Context "gets the right neurons firing" and why RAG does well. It's also why reranking with RAG improves output, in further increases the quality and/or relevance into a smaller context.

Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions

Re:Ask HN: Is RAG the Future of LLMs? by spencerchubb on Sunday April 14, 2024 @07:27PM (#40035447)
Good point! If datasets get that large that is an argument in favor of RAG