Ah, but when you can leave most of your data on the GPU side and only transfer little bits to solve your problem GPUs absolutely shine.

Some problems are well suited to this kind of architecture and some not, if you're in luck the gains can be significant and it is definitely something you should be aware of and weigh when you're writing something that is heavy on computation.