Data center / servers are as much about power usage as anything else these days. Mobile is entirely about power usage because it affects battery life, weight, and size.

These two markets combined are the vast majority of all general-purpose computing devices sold. Chip companies are following the money.

The scaling conundrum is also a factor: you must employ parallelism to get performance out of modern CPUs. GPUs are entirely about parallelism. If your problem requires scaling (ex: 3D movie render farm) you get far more bang for your buck by scaling across slower but cooler machines than you'd get with fewer machines using expensive water-cooling (and the associated A/C or central condenser). You already had to do the work to employ local parallelism... scaling across machines isn't a huge leap. If you don't go parallel the best you could hope for in your wildest fevered dream would be 2-5x perf. Scaling across cores on one machine can easily match that; across machines gives you orders of magnitude.

So even for people who require peak performance and are willing to pay millions to get it the market is cooler slower multi-core CPUs.

That leaves relatively tiny portion of the market who require maximum single-core performance because they're problem can't be paralellized. At best you could get 1-5x speed up with an over-clocked water-cooled rig and some people do. That performance factor is slowly decreasing over time as feature sizes shrink.

The IBM POWER8 goes up to 5GHz, for those few applications where single-threaded performance really matters.

IBM zEC12 goes to eleven, err 5.5 GHz.

It's not only a question of getting enough power IN but also of getting the "consumed" power(waste heat) OUT

Yes, but the point is that semiconductor companies don't focus on that problem because you'd be dissipating power anyway, which is less attractive, according to the article.