Also in practice FlashAttention is still relatively new so it isn't well supported in libraries yet. Until PyTorch 2.0 you had to either implement it yourself, or use something like xformers which comes with a bag of caveats. PyTorch 2.0 now has it built-in, and it's easy to use, but the implementation is incomplete so you can't, for example, use it with an attention mask (which is needed in LLMs, for example).
tl;dr: Basically none, but it just isn't well supported yet.
Let ๐ be the sequence length, ๐ be the head dimension, and ๐ be size of SRAM with ๐ <= ๐ <= ๐๐. Standard attention (Algorithm 0) requires ฮ(๐๐+๐ยฒ) HBM accesses, while FlashAttention (Algorithm 1) requires ฮ(๐ยฒ๐ยฒMโปยน) HBM accesses.
"standard attention has memory quadratic in sequence length, whereas FlashAttention has memory linear in sequence length."
I guess you have just reported how many times the layer will need to access the memory, not how much memory usage scales with sequence length.
mind explaining why this is so attractive/what the hurdle is for the laypeople in the audience? (me)