There is a huge amount of CUDA middleware while there is very little for OpenCL. In the case of deep learning, there's cuDNN which is highly optimized and used by all major frameworks. AMD has promised something equivalent, but a first version is very unlikely to have the same scope and features. (Nvidia probably does a large amount of dog fooding given their deep learning based automobile efforts.)

Using anything but Nvidia for deep learning is a great idea if you want to get noticed for not using Nvidia for deep learning, but it wouldn't be the smart thing to do if you want to actually get something done.

CUDA has a strong grasp on a lot of the scientific market because Nvidia was ahead of the curve with CUDA initially and then has spent tons of money advertising and selling it to entrench it. To some degree it's a dev's own fault for getting sucked into a proprietary ecosystem, but ultimately "switch your code to something else" is not always a realistic option if you want to keep up with a community where CUDA is popular.

CUDA is not as dominant as it was at first now that there are serious alternatives, but Nvidia works hard (both carrot and stick) to keep you locked in. And their hardware and software tools genuinely do have great performance, even if they should be better if Nvidia was a bit less... well evil honestly.

Yeah, that's very Nvidia-like. Hopefully Vulkan based solutions will phase out CUDA lock-in in the future.