Modern GPUs have very little specialized graphics hardware. This doesn't mean there aren't better architectures, but this idea that it's bad cause it's a graphics accelerator is horribly out of date.

GPUs were largely programmable parallel processors by the time they became in vogue for deep learning.

Bill Dally (nVidia chief scientist) has said that the hard wired graphics takes up so little die area that it is essentially negligible in terms of cost for them.

Plus, the latest Volta chips have "tensor cores" the equivalent of little TPUs.