News for hackers. Stuff that matters.
Re:Tensor Product Attention Is All You Need by hansvm on Wednesday January 22, 2025 @11:23AM (#42794455)
EA doesn't quite fit in the same umbrella. EA has a constant cache size (it's just another classical recurrent architecture inspired by approximating transformers), where this paper gives speedups to a variety of true attention mechanisms which still require caches to be proportional to the sequence length.
Re:Tensor Product Attention Is All You Need by verdverm on Wednesday January 22, 2025 @12:09PM (#42795011)
very succinct and insightful, thank you!