The bug that taught me more about PyTorch than years of using it

Re:The bug that taught me more about PyTorch than years of u... by jebarker on Sunday October 26, 2025 @10:33AM (#45712226)

This is a great write up and I’d love to see more like it. Debugging this sort of thing in the megatron->pytorch->CUDA stack is what my team spends more than half of their time on as an ML research team.

[ Parent | on HN ]

Re:The bug that taught me more about PyTorch than years of u... by ddelnano on Monday October 27, 2025 @01:49PM (#45724142)

Wouldn't the Nsight Systems suite provide coverage here? Are the tricky cases difficult to debug with the standard CUDA tooling stack?

[ Parent | on HN ]

Re:The bug that taught me more about PyTorch than years of u... by jebarker on Monday October 27, 2025 @02:17PM (#45724466)

Yes, nsys is very helpful, especially when looking at perf issues. It’s often the case that bugs present like in this blog though - you just notice that training curves have regressed somehow - so even with good tooling it can be hard to figure out where to start looking in these very complex systems. Only gets worse if the symptoms only show up when running for a long time and at scale in a cluster.

[ Parent | on HN ]