News for hackers. Stuff that matters.
Re:Generative AI and the big buzz about small language models by compressedgas on Friday March 01, 2024 @08:33PM (#39568976)
Piece with less detail than the source linked from the article: https://www.together.ai/blog/stripedhyena-7b
Re:Generative AI and the big buzz about small language models by milliondreams on Friday March 01, 2024 @05:51PM (#39567771)
As we see these systems evolving, I have come to believe specialist small language models with an MoE framework are the future of the industry.
Re:Generative AI and the big buzz about small language models by swimwiththebeat on Friday March 01, 2024 @06:11PM (#39567930)
Does anyone know if this is using the Mamba architecture[1] instead of transformers? It looks like it uses a state space model (SSM) layer.

[1]: https://arxiv.org/abs/2312.00752

Re:Generative AI and the big buzz about small language models by sal9000 on Friday March 01, 2024 @07:28PM (#39568604)
It came earlier than Mamba. It uses hyena hierarchy blocks, which are considered SSM but not the same as Mamba.
Re:Generative AI and the big buzz about small language models by milliondreams on Monday March 04, 2024 @02:15AM (#39587814)
We covered state space models in a blog post here - https://blog.dragonscale.ai/state-space-models/

It gives overview of Mamba And StrypedHyna.