Re:Mamba Explained: The State Space Model Taking On Transfor...
by ogogmad on Sunday February 25, 2024 @05:50PM
(#39505758)
OK, I've noticed that the pseudo-code above is vectorised, and so there's no recursion. The SSM function is actually described at the start of the paper, and an efficient hardware-aware implementation is suggested in section 3: https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf
Something I've noticed is that B, C and Δ depend only on the current token. See this: https://www.kolaayonrinde.com/blog/images/mamba/ssm_algorith...
+1 though for making me go back to the article and read it more carefully! +1 also to the article.