Vision Mamba: Efficient Visual Representation Learning with Bidirectional SSM

Re:Vision Mamba: Efficient Visual Representation Learning wi... by prefer_to_ask on Thursday February 01, 2024 @03:02PM (#39220589)

The innovation is that they managed to write a hardware-aware kernel to make it run fast/efficiently on GPUs. The authors of Mamba are the same authors of FlashAttention which was a performance optimization kernel written to reduce IO while computing normal O(N^2) attention. As I understand it, previously SSM models were not as easily parallelizable as the Transformer architecture.

[ Parent | on HN ]

Re:Vision Mamba: Efficient Visual Representation Learning wi... by wityl on Thursday February 01, 2024 @03:48PM (#39221234)

They were previously parallelizable (via fft), but performed poorly on language modeling tasks.

Mamba adds a dependence on the inputs that makes language modeling competitive with transformers, but that prevents using the fft approach. So they switch to a method using parallel prefix scan.

[ Parent | on HN ]

Re:Vision Mamba: Efficient Visual Representation Learning wi... by kelseyfrog on Thursday February 01, 2024 @03:47PM (#39221213)

Yes and no. There's an dual connection between ssms and convolutional models if certain constraints are met. Training convolutionally and inferring sequentially seeks a compromise between the two sides. I think we're about to find out the degree to which those constraints impact "easily".

[ Parent | on HN ]