News for hackers. Stuff that matters.
Re:Stability AI Launches the First of Its StableLM Suite of ... by sroussey on Wednesday April 19, 2023 @04:54PM (#35633198)
So there must be a downside to FlashAttention. What is it?
Re:Stability AI Launches the First of Its StableLM Suite of ... by lhl on Wednesday April 19, 2023 @05:36PM (#35633619)
https://arxiv.org/abs/2205.14135 - Section 5 suggests that the biggest limitation is that custom CUDA kernels need to be coded on a per-GPU architecture basis.
Re:Stability AI Launches the First of Its StableLM Suite of ... by fpgaminer on Wednesday April 19, 2023 @05:39PM (#35633640)
FlashAttention is mathematically identical to standard attention, so in theory there's no downside. In practice, numerical inaccuracies of floating point mean that the results differ slightly. I don't know of any papers going in depth to analyze what impact those variances have in a range of real models, but generally speaking deep models handle slightly variances well. I've not noticed any difference in my applications training models. And tons of people use FlashAttention as a drop-in replacement on models trained on standard attention (e.g. using xformers in StableDiffusion).

Also in practice FlashAttention is still relatively new so it isn't well supported in libraries yet. Until PyTorch 2.0 you had to either implement it yourself, or use something like xformers which comes with a bag of caveats. PyTorch 2.0 now has it built-in, and it's easy to use, but the implementation is incomplete so you can't, for example, use it with an attention mask (which is needed in LLMs, for example).

tl;dr: Basically none, but it just isn't well supported yet.

Re:Stability AI Launches the First of Its StableLM Suite of ... by kiraaa on Wednesday April 19, 2023 @06:30PM (#35634137)
installing it is a nightmare