Writing Speed-of-Light Flash Attention for 5090 in CUDA C++

Re:Writing Speed-of-Light Flash Attention for 5090 in CUDA C++ by neilmovva on Saturday August 23, 2025 @05:19PM (#44999232)

Today, training in "low precision" probably means computing FP8 x FP8 -> FP32. The FP32 accumulation is still important, but otherwise yes this works, especially if we're talking about MXFP8 as supported on Blackwell [0].

What's less proven is a recipe using MXFP4 x MXFP4 -> FP32 compute, e.g. [1], which needs more involved techniques to work. But if you get it to work stably, that pathway is running at full throughput on 5090.

[0]: https://arxiv.org/abs/2506.08027 [1]: https://arxiv.org/abs/2502.20586

[ Parent | on HN ]

Re:Writing Speed-of-Light Flash Attention for 5090 in CUDA C++ by laidoffamazon on Saturday August 23, 2025 @06:11PM (#44999543)

Interesting. My assumption was one of the innovations of DeepSeek and the modern GPT models was performing low precision pretraining rather than just finetuning further. I didn't realize you still need accumulation at a higher precision anyway

[ Parent | on HN ]