News for hackers. Stuff that matters.
Re:Mamba Explained: The State Space Model Taking On Transfor... by ogogmad on Sunday February 25, 2024 @03:47PM (#39504567)
That might explain the motivation for why the Δ variable is used and varied; but not the "Selectivity", which the article says is expressed by how the matrices B and C vary while consuming input.

Something I've noticed is that B, C and Δ depend only on the current token. See this: https://www.kolaayonrinde.com/blog/images/mamba/ssm_algorith...

+1 though for making me go back to the article and read it more carefully! +1 also to the article.

Re:Mamba Explained: The State Space Model Taking On Transfor... by ogogmad on Sunday February 25, 2024 @05:50PM (#39505758)
OK, I've noticed that the pseudo-code above is vectorised, and so there's no recursion. The SSM function is actually described at the start of the paper, and an efficient hardware-aware implementation is suggested in section 3: https://arxiv.org/ftp/arxiv/papers/2312/2312.00752.pdf