Deep Neural Networks involve Convolution and Matrix Multiplication operations which have a well defined hard upper bound in terms of speed wrt number of cores, GPUs (~thousands of processors) are in thus indispensable achieving a faster performance. There are newer interesting approaches such as reducing multiplication operations which might improve inference/forward speed even further. http://arxiv.org/abs/1510.03009