How it compares to the many hybrid SSM/Transformer architectures that are coming out now, with 2B+ weights released? I skimmed the paper, but it seems it's innovation is basically combining both in the same layer, instead of alternating layers, but I see no reference to the latter.

Maybe a bit outdated now, but reminds me of LSTMs from the recurrent update of a memory / hidden state with gating. I remember one of the biggest problems with such RNNs being vanishing gradients as a result of the long context, which vanilla transformers presumably avoided by parallellizing over the context instead of processing them individually. I wonder how this is avoided here?

"Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention" (2024) https://arxiv.org/abs/2404.07143 :

> This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

It is missing a proper evaluation section. Specifically, I wonder how this compares with https://arxiv.org/abs/2401.03462 (Activation Beacon)