A 100 million token context that takes an hour to start returning an answer to a prompt isn't very useful for most things.
As long as there is a relationship between the length of the context and the time it takes to produce an output, there will be a reason to be selective about what goes into that context - aka a reason to use RAG techniques.
Which is all to say really, data quality is probably the most distinguishing factor in LLM/RAG systems... to get back to OP's questions
Rag is not bad attention, rag is the user not knowing the context to give the LLM
“This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation”
I could very well be wrong! But we wouldn't want LLMs to be performing lots of arithmetic calculations via exploiting hidden parts of themselves that do linear regression or whatever, far better to just give them the calculator and get results faster and cheaper. Similarly, we can give them a search engine (RAG) and let them figure it out more efficiently.
I'd assume "large enough" context is the actual goal here, not "virtually infinite".
I think LLM context is going to be like cache levels. The first level is small but super fast (like working memory). The next level is larger but slower, and so on.
RAG is basically a bad version of attention mechanisms. RAG is used to focus your attention on relevant documents. The problem is that RAG systems are not trained to minimize loss, it is just a similarity score.
Obligatory note that I could be wrong and it's just my armchair opinion