I just want to support one of your points: for some applications, the PCIe latency kills the potential of using GPUs. It doesn't matter how fast the GPU is at a computation if the time to transfer the data is longer than it would take to do the calculation on the host CPU.

I published a paper with this conclusion a while back ("Evaluation of Streaming Aggregation on Parallel Hardware Architectures", http://www.scott-a-s.com/files/debs2010.pdf). My paper was basically, "GPUs are awful for our problem, so let's compare to CPUs and Cell". A more recent paper from NSDI 2015 from a group at CMU, with Anuj Kalia as the first author, that also looks into it, "Raising the Bar for Using GPUs in Software Packet Processing", http://www.cs.cmu.edu/~akalia/doc/nsdi15/gopt_nsdi.pdf. They look at just the GPU and CPU part of the spectrum - which is probably more relevant to most, as the Cell is a dead technology now. (I think my conclusions will matter more when on-chip GPUs become more prevalent, but we're not there yet). Their paper starts with an excellent analysis of the latencies for transferring data to GPUs, and comparing end-to-end performance against several different algorithms, where they vary the amount of data reuse.

My consistent gripe when talking about improving application performance by using accelerators is that one must take data copying costs into consideration, which means we must use application end-to-end performance as our metric. There's a temptation to focus on just the computational kernel performance. The Kalia et al. paper is the best treatment of this subject that I have seen.