GPUs give you great throughput, but they're expensive, eat a lot of power, only work well in batch and aren't tuned to prediction (eg poor INT8 performance and too much VRAM).