News for hackers. Stuff that matters.
Re:Show HN: Flash Attention in ~100 lines of CUDA by xrd on Saturday March 16, 2024 @04:43PM (#39729333)
Any CUDA primer you recommend in particular? I had this same question.
Re:Show HN: Flash Attention in ~100 lines of CUDA by winwang on Saturday March 16, 2024 @05:56PM (#39729842)
Here's an article on syncing in CUDA via cooperative groups: https://developer.nvidia.com/blog/cooperative-groups/

There's also explicit warp synchronization, i.e. __syncwarp(). More on warp primitives here: https://developer.nvidia.com/blog/using-cuda-warp-level-prim...

Re:Show HN: Flash Attention in ~100 lines of CUDA by cavisne on Saturday March 16, 2024 @07:02PM (#39730231)
Probably https://www.youtube.com/watch?v=nOxKexn3iBo (or just skimming the attached colab).
Re:Show HN: Flash Attention in ~100 lines of CUDA by xrd on Saturday March 16, 2024 @07:38PM (#39730435)
This is terrific, thanks!