News for hackers. Stuff that matters.
Re:Stability AI Launches the First of Its StableLM Suite of ... by WithinReason on Wednesday April 19, 2023 @06:07PM (#35633930)
According to the paper Flash Attention also needs quadratic memory:

Let ๐‘ be the sequence length, ๐‘‘ be the head dimension, and ๐‘€ be size of SRAM with ๐‘‘ <= ๐‘€ <= ๐‘๐‘‘. Standard attention (Algorithm 0) requires ฮ˜(๐‘๐‘‘+๐‘ยฒ) HBM accesses, while FlashAttention (Algorithm 1) requires ฮ˜(๐‘ยฒ๐‘‘ยฒMโปยน) HBM accesses.

Re:Stability AI Launches the First of Its StableLM Suite of ... by GaggiX on Wednesday April 19, 2023 @06:45PM (#35634258)
https://github.com/HazyResearch/flash-attention#memory

"standard attention has memory quadratic in sequence length, whereas FlashAttention has memory linear in sequence length."

I guess you have just reported how many times the layer will need to access the memory, not how much memory usage scales with sequence length.