Re:Stability AI Launches the First of Its StableLM Suite of ...
by GaggiX on Wednesday April 19, 2023 @06:45PM
(#35634258)
https://github.com/HazyResearch/flash-attention#memory
"standard attention has memory quadratic in sequence length, whereas FlashAttention has memory linear in sequence length."
I guess you have just reported how many times the layer will need to access the memory, not how much memory usage scales with sequence length.
Let ๐ be the sequence length, ๐ be the head dimension, and ๐ be size of SRAM with ๐ <= ๐ <= ๐๐. Standard attention (Algorithm 0) requires ฮ(๐๐+๐ยฒ) HBM accesses, while FlashAttention (Algorithm 1) requires ฮ(๐ยฒ๐ยฒMโปยน) HBM accesses.