It may be computation-bound (I'm not sure) but training deep networks generally does use a lot of memory, because of the giant training sets. You're right that GPUs are a good fit, for example libraries such a theano exploit this.