It's just a historical fact that GPUs originally evolved to do graphics. What to they're really good at is doing a huge amount of numerical computation in parallel, of which graphics is just a specific use case, and is exactly what you need for ML. If they were invented today, they would be called something else. They're already the specialized hardware you're describing.