News for hackers. Stuff that matters.
Re:Mamba: The Easy Way by magnio on Friday February 23, 2024 @12:56PM (#39483911)
Fantastic blog post, thank you for this. I am not even familiar with transformers, yet the explanation is stellar clear to me, and the included references and context are a trasure trove. The explanation of FlashAttention is the best I have seen, and that is not even the focus of the article.

One question I have on selectivity: footnote 4 says "the continuous A is constant, while our discretization parameter ∆ is input-dependent." What is the effect of varying the discretization instead of the (main, as I understand it) state A? My gut says it simplifies training and provides stability, but I feel A carries most of the behavior of the model, so it should have more wiggle room throughout training.

Re:Mamba: The Easy Way by jackcook on Friday February 23, 2024 @01:10PM (#39484095)
Thank you for the kind words! I think it’s mostly to reduce complexity during training. Here’s an excerpt from page 9 of the Mamba paper:

“We remark that while the A parameter could also be selective, it ultimately affects the model only through its interaction with ∆ via A = exp(∆A) (the discretization (4)). Thus selectivity in ∆ is enough to ensure selectivity in (A, B), and is the main source of improvement. We hypothesize that making A selective in addition to (or instead of) ∆ would have similar performance, and leave it out for simplicity.”

Re:Mamba: The Easy Way by nlrk on Saturday February 24, 2024 @01:14AM (#39489544)
when I read the paper I thought the idea was changing \Delta permits getting the model to learn different things over different time scales. As you quoted “the main source of improvement".

I don’t have an llm backround, just controls, so I might wrong.

Re:Mamba: The Easy Way by whimsicalism on Friday February 23, 2024 @01:39PM (#39484465)
How are you not familiar with transformers yet have seen multiple explanations of FlashAttention?
Re:Mamba: The Easy Way by samus on Friday February 23, 2024 @04:27PM (#39486334)
The issue with Attention essentially is that it is used to relate all token of the input sequence with each other. The need to do that somehow makes sense no matter how much one understands about the internals of a transformer. The naive way to do that boils down to matrix multiplications, and a lot more people understand the performance issues implied by them.
Re:Mamba: The Easy Way by whimsicalism on Friday February 23, 2024 @05:10PM (#39486783)
your comment makes no sense to me, sorry. if you understand attention you understand transformers, period.
Re:Mamba: The Easy Way by defrost on Friday February 23, 2024 @07:56PM (#39488098)
Likewise your comment(s) makes no sense to me.

If you can understand attention and transformers, how can you not understand that population numbers can rise, reach a peak, fall, and then level out (all w/out any genocidial actions)?

How can you claim that it is "absurdism" to imagine something that can be seen in data across the plant and animal kingdom?

Re:Mamba: The Easy Way by samus on Friday February 23, 2024 @05:39PM (#39487059)
That's good to know :)
Re:Mamba: The Easy Way by avarun on Friday February 23, 2024 @02:55PM (#39485328)
Literally the exact question I had reading that comment haha