News for hackers. Stuff that matters.

A comment on Show HN: Flash Attention in ~100 lines of CUDA.

Re:Show HN: Flash Attention in ~100 lines of CUDA by danielhanchen on Saturday March 16, 2024 @10:58PM (#39731535)
Fantastic work! Extremely neat and clear implementation! Interesting note on the backward pass - what do you think are the main blockers for a backward pass?
[ Parent | on HN ]
Re:Show HN: Flash Attention in ~100 lines of CUDA by tspeterkim on Sunday March 17, 2024 @05:59AM (#39733204)
Thanks Daniel. The main blocker is me not able to fully grasp the backward pass. (trying to understand Appendix B.2 in the original paper)

I need to get more comfortable with matrix derivatives before I can confidently reimplement it in the same minimal way as I did with the forward pass.

[ Parent | on HN ]
Re:Show HN: Flash Attention in ~100 lines of CUDA by danielhanchen on Sunday March 17, 2024 @08:57AM (#39734187)
Oh ok! Ye the backwards passes are always much more difficult due to the derivatives!
[ Parent | on HN ]