Re:Show HN: Flash Attention in ~100 lines of CUDA
by xrd on Saturday March 16, 2024 @04:43PM
(#39729333)
Any CUDA primer you recommend in particular? I had this same question.
Re:Show HN: Flash Attention in ~100 lines of CUDA
by winwang on Saturday March 16, 2024 @05:56PM
(#39729842)
Here's an article on syncing in CUDA via cooperative groups: https://developer.nvidia.com/blog/cooperative-groups/
There's also explicit warp synchronization, i.e. __syncwarp(). More on warp primitives here: https://developer.nvidia.com/blog/using-cuda-warp-level-prim...
Re:Show HN: Flash Attention in ~100 lines of CUDA
by cavisne on Saturday March 16, 2024 @07:02PM
(#39730231)
Probably https://www.youtube.com/watch?v=nOxKexn3iBo (or just skimming the attached colab).
Since the threads are relying on each other to fill the SRAM with all needed data if you didn’t wait then values would be missing.