>If someone could figure out a way to write regular software that easily parallelized to hundreds of cores, CPU's could also get the same leaps in performance that GPU's have been enjoying. It's a software problem more than it is a hardware problem.

Amdahl's law [1] , and while we have demonstrate parallelised code can speed things up, generally speaking Software performance still have lots of low hanging fruit without even going deep into parallel and concurrency world, it is just a matter of cost whether it is worthwhile to improve or fix it.

[1] https://en.wikipedia.org/wiki/Amdahl%27s_law