Re:Rag Is a Hack
by ldjkfkdsjnv on Friday October 06, 2023 @03:26PM
(#37795099)
Isn't RAG analogous to how humans get information during a conversation or searching the internet? Its contextual world information, in the future RAG will also include vision, hearing, sight, the rest of the senses etc. But those are just lower level inputs on what the current language inputs are
At a high level yes? There is some representation of facts in the brain, and the same is true in a language model that uses a knowledge base. But if you look any closer the differences are pretty clear. We don’t store facts as a serialized list in our brains. We have a knowledge graph, and a lossy one at that.
I see these two claims as orthogonal. AGI need not work like the human brain, and I’m not sure people’s brains have any lossless representation of memories like a RAG system would. People are infamously unreliable when it comes to recalling information during crimes for example.
However, it does seem like a RAG system is much more versatile than continuously training. Facts change all the time and it’s infeasible to retrain every time that happens.