I've been wondering the same thing. It is why I don't like the fact that we need increasingly specialized types of computers to do ML. This is also why I am very interested in research that is not based on deep neural networks, which seem to be especially hungry in terms of training data and hardware power.