I think Intel is probably planning to unify CPU and GPU. Intel has been bringing GPU-like features to CPUs like very wide SIMD, gather (vectorized memory load), etc.
Just scatter (vectorized store), a 2D-optimized cache controller (for trilinear filtering and hardware swizzling) and fast HBM memory are missing from the latest Skylake CPUs. Otherwise current crop of CPUs can pretty much be a GPU. An 18 core 2.3 GHz Haswell Xeon has 1.3 TFLOPS peak.
> CPUs are WAY faster than most people realize but the way current software is written leaves anywhere from 10x-100x the performance on the table.
Yeah, just a single Haswell core can do 32 (FMA) flops per clock cycle. Other then specialty optimized software, it might see a flop or two every few cycles while being idle a lot.
On the other hand, most software can't be 10-100x faster. 2-10x is pretty common, though. Compilers are still somewhat bad at using SIMD cleverly. Sometimes they do surprise.
> Right now the best option is ISPC, Intel's compiler.
Right now the best option is to still to use assembler or intrinsics (when compiler does ok job at instruction scheduling), at least if you need serial performance in "normal" applications. ISPC is of course nice for very parallel problems.
How do I know? We have some experience with vectorization in the team I work in (www.gromacs.org), our code has support for at about 10-12 SIMD instruction sets ;)
Compiler vectorization tends to also be less stable, slightest changes can make the compiler to drop the optimizations in a later build.
It might not ideal to use intrinsics, but that's how you get predictable performance.
Not even close, but you wouldn't know it from looking at Intel. They have 18 core Xeon processors that don't need to cost $4k USD. They are planning a knight's landing Xeon Phi with 72 out of order cores, each with 4 hyperthreads and two 512 bit SIMD units (meaning 1 core = 32 float operations at a time!). It will also have special high bandwidth memory for a 16GB cache in between the CPUs and real memory.
Intel's consumer chips have led to articles like this. Part is zero competition from AMD, part is intense competition from ARM and everyone else.
GPUs have not gone without significant improvements. 5 years ago the 580 gtx might have been the best GPU.
The other problem is that there isn't a heavy demand for faster CPUs on the desktop. Software isn't being written with multiple cores in mind, and if it is, it makes use of only 2 or 4 cores for a limited part of the program.
All of this is on top of the fact that the latency from memory is not getting faster. CPUs are WAY faster than most people realize but the way current software is written leaves anywhere from 10x-100x the performance on the table. From the hardware angle, higher clock speeds don't add anything.
Then you get into issues of cache coherent multi-threaded software being very difficult to write for the vast majority of people. How do you use SIMD? Right now the best option is ISPC, Intel's compiler. Everything else is a hail mary to the compiler. How do you use multiple threads? There are various wheels, none of them round. Even allocating memory so that you avoid a system global lock is an issue far deeper than most people will go when writing software.
Intel is in a position where they would look really much better if software was more optimal, but at the same time the fact that it isn't is part of the reason they have crushed their competition in the past.