Memory bandwidth is the issue for scaling CPU cores. Very, very few problems can saturate 32 cores worth of computational work with the N-channel DDR memory, so there's no reason to design a Xeon or Opteron with more than that many. There may be challenges related to N-way cache coherency too.

> memory bandwidth limits mean the difference between optimized CPU, GPU and FPGA systems is not that much.

Well, not really -- the enormous memory throughput of GDDRx is much of what's fueled the growth in the GPGPU market over the last decade.

It totally depends on your algo. If you play to the GPUs strengths it will own CPU through pure additional grunt, but you have to design for it.