Offloading to the GPU has a significant cost that needs to be amortized. DNNs work well because very little is moved over the PCIe bottleneck, and inputs can be buffered. Take modern compression, let's say H265, has a complex control flow combined with highly vectorizable work. I'm unsure where the threshold lies today, but you need a significant amount of work before offloading to and reading back from the GPU becomes interesting on a beefy Xeon.