It's just a historical fact that GPUs originally evolved to do graphics. What to they're really good at is doing a huge amount of numerical computation in parallel, of which graphics is just a specific use case, and is exactly what you need for ML. If they were invented today, they would be called something else. They're already the specialized hardware you're describing.

> Wouldn’t it make more sense to let the CPU do it, or have specialized cards for the job. So you could have an AMD graphics card and an Nvidia ML card.

Graphics cards are the specialized cards for ML. They are several orders of magnitude faster than GP CPUs at ML because they are specifically designed for the mathematical calculations done in most ML.

We call them graphics cards because that was the first real general use-case for doing lots of linear algebra on large data sets. But long before CUDA came along, people were abusing graphics API to perform more general purpose high-performance calculations for things like video decompression or physics.

nVidia has good leadership and recognized that there would soon be a sizeable market for video cards as a cheap replacement for specialized hardware used in super computers.

Interesting tidbit, when I was in college, one of our CS professors invested in a cluster of PS3s as a cheap "super computer" as they were substantially faster at ML than anything else you could get for $600.

GPUs and also the specialized cards basically can do the same vector, matrix and tensor operations that are needed for neural network machine learning.

CPUs are not faster, in fact they are several orders of magnitude slower for these operations.