GPUs have both pipelining and caches, so those don't serve to distinguish your idea from a GPU. Also remember that the barrier to spreading out computation over more cores is that many of our algorithms don't parallelize well, not that we can't fit 1024 cores into a computer. There's already been a huge incentive to get programs (even those used on regular desktops and not high performance computers) to parallelize well since we hit the clock-rate ceiling, but a lot of problems still have trouble saturating even a 4 core system.