GPUs and also the specialized cards basically can do the same vector, matrix and tensor operations that are needed for neural network machine learning.

CPUs are not faster, in fact they are several orders of magnitude slower for these operations.