Re:Mixtral 8x22B
by samus on Wednesday April 17, 2024 @04:35PM
(#40069794)
So far, people have come up with many alternatives for quadratic attention. Only recently have they proven their potential.
[ Parent
| on HN ]
Re:Mixtral 8x22B
by underlines on Wednesday April 17, 2024 @05:01PM
(#40070022)
tons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too:
https://arxiv.org/html/2404.08801v1 Meta Megalodon
https://arxiv.org/html/2404.07143v1 Google Infini-Attention
https://arxiv.org/html/2402.13753v1 LongRoPE
and a ton more