Memory bandwidth is the issue for scaling CPU cores. Very, very few problems can saturate 32 cores worth of computational work with the N-channel DDR memory, so there's no reason to design a Xeon or Opteron with more than that many. There may be challenges related to N-way cache coherency too.

> memory bandwidth limits mean the difference between optimized CPU, GPU and FPGA systems is not that much.

Well, not really -- the enormous memory throughput of GDDRx is much of what's fueled the growth in the GPGPU market over the last decade.