That depends on the problem space. For some problems latency does not need to be an issue, if the problem can be pipe-lined then you can keep all your cores running at a high percentage of their theoretical maximum load even if latency is high.
For real time applications this is usually not an option, there latency > some threshold will kill your proposed solution.
In those cases where the latency is not a problem, it tends to be precisely because you can hide it e.g. via pipelining :)
While GPU offloading is extremely powerful in terms of compute, you have to deal with both bandwidth limitations (de facto ~12 GB/s for x16 PCIe 3.0) and the latency of launching compute kernels and waiting for them to complete.