Re:Attention at Constant Cost per Token via Symmetry-Aware T...
by cubefox on Wednesday February 04, 2026 @10:43AM
(#46887212)
I think DeepSeek V3.2 is sub n^2, but it clearly performs quite well, refuting the alleged lower bounds in the paper.
[ Parent
| on HN ]
Re:Attention at Constant Cost per Token via Symmetry-Aware T...
by andy12_ on Wednesday February 04, 2026 @11:13AM
(#46887628)
It really isn't sub N^2. The main attention is only O(Nk), but only thanks to a lightning indexer that still has complexity O(N^2). So overall it still has the same complexity; just with a smaller constant factor [1]
> DSA reduces the core attention complexity of the main model from O(L^2) to O(Lk), where k (<< L) is the number of selected tokens. Although the lightning indexer still has a complexity of O(L^2), it requires much less computation compared with MLA in DeepSeek-V3.1-Terminus
Re:Attention at Constant Cost per Token via Symmetry-Aware T...
by cubefox on Wednesday February 04, 2026 @01:03PM
(#46889229)
Okay, then let's see whether we are going to see real linear architectures, like Gated DeltaNet or Mamba-3, in some larger models. I don't believe there is a "lower bound" which states that those can never get to (or exceed) the real-world performance of quadratic attention. (Perfect recall in unrealistic needle-in-haystack tests doesn't count.)