CUDA, OpenCL, Vulkan, and Metal are all different graphics APIs provided by GPU vendors. GPU vendors can implement all four of them if they like, but in practice, CUDA is an NVidia exclusive, Metal is a Apple exclusive and vice versa, and OpenCL is kind of old and unused (and also used to be an Apple technology as well). There's also other APIs provided by non-GPU hardware vendors.

PyTorch and TensorFlow are training and inference libraries. In theory, PyTorch at least (probably also TF) is properly abstracted and can work with all APIs. In practice CUDA gets the most testing and has all the operator fusion kernels you need for good performance, so everyone trains on CUDA. Inference is less resource-intensive than training, and there's a lot of vendors that provide inference-only hardware (e.g. "Edge TPUs") that's better supported, so on the inference end there's more variety in API usage.

You can of course train and inference on CPU but that means terrible performance.

CUDA and OpenCl are not a graphics APIs, they're designed for computing only.

Devices like nVidia H100 and AMD MI250 are also designed for computing only.

They don’t support D3D, GL or Vulkan, yet both manufacturers still call these things “a GPU”.

Well, you're basically correct that the ML GPU libraries basically all build on CUDA with non-Nvidia being a second class citizen, but then stuff like OpenGL, Vulkan, DirectX and Metal are more general "os-level" or low level GPU graphics and compute abstractions. This, and the popular rust project `wgpu` which is the base for Firefox's WebGPU implementation abstract over these lower-level abstractions that may or may not be available such that the same, often a bit simpler code is usable accross multiple types of devices. Basically the opposite of needing CUDA.

Nope, they don’t all build on CUDA. OpenCL and Vulkan are cross-platform standards developed by Khronos group, the same as who developed OpenGL. OpenCL aimed to be roughly the equivalent of CUDA, but in practice the developer experience of CUDA has far outpaced it. Vulkan is targeted at graphics, unlike CUDA. Metal is a similar API to Vulkan, i.e. a low-overhead graphics API targeted at modern GPUs, but only on Apple platforms. PyTorch and Tensorflow are high-level APIs targeted at deep learning, which have multiple execution backends, one of the most popular being CUDA, but they can also run on CPUs or other processors such as TPUs.

When innovative hardware/firmware introduces a new feature that wasn't captured in a previous hardware abstraction layer (if any such HAL exists for that class of hardware), the manufacturer will somehow need to make that feature available to their customers.

Depending on the scope of the feature and the investment the manufacturer wants to make, that's going to be through their driver, through some opaque handle-passing API in the HAL, or through a whole SDK. This not only lets them show off their new functionality in a way that they wouldn't be able to do otherwise, but gives them an opportunity to "lock in" customers (as NVidia did quite effectively with CUDA).

Between crypto, ML, and the already-immense gaming industry, a huge amount of money has gone into "graphics" hardware over the last 10+ years and it's led to lots of novel innovations. And since those innovations weren't anticipated by existing HAL's like DirectX and OpenGL, there's a lot of these new manufacturer-specific tools as well as numerous attempts to fight "lock in" by wrangling them into new repurposable abstractions.

It's just a cyclical process. It'll gradually pass as innovations in this sector plateau again, but that still looks to be some ways out.

You are mixing up several levels of abstraction. It probably helps to realize that the GPU accelerated ML stuff was originally built on top of dedicated graphics libraries/APIs but diverged as it has different needs. You have here three somewhat distinct groups of things, but the boundaries are a bit fuzzy.

Graphics interfaces: Vulcan, Metal ( and older OpenGL, D3D) Non-graphics GPU Compute interfaces: CUDA/(and RocM etc.) Computational Frameworks: Pytorch/Tensorflow/etc.

The graphics interfaces are somewhat distinct from the ML/compute stuff, but as they use the same kind of underlying hardware to do work it's not that simple.

The frameworks are trying to abstract computations from the underlying hardware, e.g. you can run your tensorflow/pytorch code against CPU only, etc. In practice though for "serious" work NVIDIA gpus are the likely target.

The GPU interfaces are trying to do the same thing, roughly, but NVIDIA is far ahead in terms of hardware and dev support, so mostly that is what is deployed.