News for hackers. Stuff that matters.

A comment on Show HN: Flash Attention in ~100 lines of CUDA. Parent comment

Re:Show HN: Flash Attention in ~100 lines of CUDA by tspeterkim on Sunday March 17, 2024 @05:59AM (#39733204)
Thanks Daniel. The main blocker is me not able to fully grasp the backward pass. (trying to understand Appendix B.2 in the original paper)

I need to get more comfortable with matrix derivatives before I can confidently reimplement it in the same minimal way as I did with the forward pass.

[ Parent | on HN ]
Re:Show HN: Flash Attention in ~100 lines of CUDA by danielhanchen on Sunday March 17, 2024 @08:57AM (#39734187)
Oh ok! Ye the backwards passes are always much more difficult due to the derivatives!
[ Parent | on HN ]