Yes, one is committing knowledge to neurons, the other is commuting knowledge to short term memory.

Put another way, if you took a llm with random weights. Do you expect you could rely on context alone?