Performing inference with deep learning modes is computationally costly as well. Its a reason why NVidia has two streams of business, Tesla / Titan / upcoming Volta GPUs to train models and Tegra boards to enable near real time inference.