News for hackers. Stuff that matters.
Re:Mixtral 8x22B by creshal on Wednesday April 17, 2024 @11:30AM (#40066119)
Wasn't there a paper yesterday that turned context evaluation linear (instead of quadratic) and made effectively unlimited context windows possible? Between that and 1.58b quantization I feel like we're overdue for an LLM revolution.
Re:Mixtral 8x22B by samus on Wednesday April 17, 2024 @04:35PM (#40069794)
So far, people have come up with many alternatives for quadratic attention. Only recently have they proven their potential.
Re:Mixtral 8x22B by underlines on Wednesday April 17, 2024 @05:01PM (#40070022)
tons and tons of papers, most of them had some disadvantages. Can't have the cake and eat it too:

https://arxiv.org/html/2404.08801v1 Meta Megalodon

https://arxiv.org/html/2404.07143v1 Google Infini-Attention

https://arxiv.org/html/2402.13753v1 LongRoPE

and a ton more