Re:Attention at Constant Cost per Token via Symmetry-Aware T...
by cubefox on Wednesday February 04, 2026 @01:03PM
(#46889229)
Okay, then let's see whether we are going to see real linear architectures, like Gated DeltaNet or Mamba-3, in some larger models. I don't believe there is a "lower bound" which states that those can never get to (or exceed) the real-world performance of quadratic attention. (Perfect recall in unrealistic needle-in-haystack tests doesn't count.)
[ Parent
| on HN ]