Re:Tensor Product Attention Is All You Need
by hansvm on Wednesday January 22, 2025 @11:23AM
(#42794455)
EA doesn't quite fit in the same umbrella. EA has a constant cache size (it's just another classical recurrent architecture inspired by approximating transformers), where this paper gives speedups to a variety of true attention mechanisms which still require caches to be proportional to the sequence length.
[ Parent
| on HN ]