Training a deep network using gradient descent is what takes all the compute. Actually using the trained network takes far, far less compute, and can be done on home computers with a good video card.

I suspect training algorithms will get better over time, however, the ambitions of models to train will likely keep things constrained for the next decade or so.