I'm not so sure anymore. The big reason is that now that the ML framework ecosystem has fragmented into different "layers" of the stack, very few people are directly writing CUDA kernels anymore.

As a result, with things like XLA now supporting AMD GPUs using RoCM under the hood the feature gap has closed A LOT.

Sure, Nvidia still has the performance crown lead with CuDNN, NCCL, and other libraries providing major boosts. But AMD is starting to catch up quite fast.