But it will probably be rejected. The quality bar for NeurIPS is quite high.

Some reasons:

The experiments are very weak: There are just a few figures, basically figure 1 and figure 5, which show some results. There are no tables with numbers.

But more importantly: There are no comparisons (in terms of experiments/numbers) to similar models, like:

- Block-Recurrent Transformers (https://arxiv.org/abs/2203.07852) and related approaches to make the Transformer recurrent, so effectively getting infinite context length.

- All the work on sparse attention, or linear attention, like Longformer, etc, which should also allow for such context lengths.

I don't mean that they just mention them in related work (they partly do, although it looks very short, so I'm quite sure they leave out a lot of other related work, without looking too much into it). I mean that they really run experiments and show numbers. And also, to look at the model definitions of those alternatives, and compare and analyze the differences.

So, while the presented approach could be interesting, it's hard to tell how good it really performs, and how it compares to alternatives.

(This is now a 10 min review by me. If I had to review this, I usually would spend more like 1-2h at least, so giving much more details. But I don't expect that my initial impression would change too much.)