> RAG is much closer to how the human brain works than fine tuning a model is.
[citation needed]
Obviously, the human brain continuously integrates new material, but whether that's more like RAG, continuous fine-tuning, just having a really big context window, or just having a completely different model architecture where activiation (or not) of pathways in inference also produces weight adjustments directly — or doesn't meaningfully work like any of those — is another question, where the answer is less obvious.
I think RAG (storing exact data losslessly with some sort of indexing to allow retrieval, and something hidden-prompt-like directing the use of that in populating an inference context) is a lot less likely that something that involves dynamically adjusting model weights. Which also explains forgetting, not just learning.
[citation needed]
Obviously, the human brain continuously integrates new material, but whether that's more like RAG, continuous fine-tuning, just having a really big context window, or just having a completely different model architecture where activiation (or not) of pathways in inference also produces weight adjustments directly — or doesn't meaningfully work like any of those — is another question, where the answer is less obvious.
I think RAG (storing exact data losslessly with some sort of indexing to allow retrieval, and something hidden-prompt-like directing the use of that in populating an inference context) is a lot less likely that something that involves dynamically adjusting model weights. Which also explains forgetting, not just learning.