News for hackers. Stuff that matters.

A comment on Mamba Explained: The State Space Model Taking On Transformers. Parent comment

Re:Mamba Explained: The State Space Model Taking On Transfor... by CuriouslyC on Sunday February 25, 2024 @02:01PM (#39503667)
This. Hyperparameter tuning and training include a lot of model specific black magic. Transformers have had time to mature, it'll take a while for other stuff to catch up even if they have a higher potential ceiling.
[ Parent | on HN ]
Re:Mamba Explained: The State Space Model Taking On Transfor... by koayon on Sunday February 25, 2024 @02:45PM (#39504083)
Definitely agree that a lot of work going into hyperparameter tuning and maturing the ecosystem will be key here!

I'm seeing the Mamba paper as the `Attention Is All You Need` of Mamba - it might take a little while before we get everything optimised to the point of a GPT-4 (it took 6 years for transformers but should be faster than that now with all the attention on ML)

[ Parent | on HN ]
Re:Mamba Explained: The State Space Model Taking On Transfor... by koayon on Sunday February 25, 2024 @02:46PM (#39504087)
Another interesting one is that the hardware isn't really optimised for Mamba yet either - ideally we'd want more of the fast SRAM so that we can store more larger hidden states efficiently
[ Parent | on HN ]