That also means that you can really only emulate as many cores as the NVidia card has streaming multiprocessors, not as many as it has shared processors or "cores".
Also, it's true that GPUs have huge memory bandwidth they achieve that by trading off against memory latency. You can actually think of GPUs as throughput optimized compute devices and CPUs as latency optimized compute devices and not be very mislead.
So I expect that the single threaded performance of a NVidia general purpose computer to be very low in cases where the memory and branch patterns aren't obvious enough to be predictable to the compiler. Not unusably slow but something like the original Raspberry Pi.
Each emulated core would certainly have very good SIMD support but at the same time pretending that they're just SIMD would sacrifice the extra flexibility that NVidia's SIMT model gives you.
There are clever ways around this limitation, see links in my post this thread.
Not totally dissimilar to the thinking behind NetBurst which seemed to be all about having a deep pipeline and keeping it fed with quality predictions.
But take a look at the MOG code and see what you can do.
Check out H. Dietz' stuff. Links above.
However, it is quite feasible to emulate, on a GPU, a networked group of a general purpose CPUs( ie, run MIMD[1] programs on SIMD[2] architecture). This, MOG[3], has been a project of Henry G. Dietz of the University of Kentucky. Unfortunately, the project seems to have stalled at a "rough" level. He claims that he can run MIMD programs at 1/4 efficiency while also running SIMD programs at near full efficiency. His video is instructive [4].
Edit: Note that this isn't intended for deep learning applications as such but rather for traditional supercomputing applications (weather prediction, other physics simulations ,etc).
[1] https://en.wikipedia.org/wiki/MIMD [2] https://en.wikipedia.org/wiki/SIMD [3] http://aggregate.org/MOG/ [4] https://www.youtube.com/watch?v=FZ6efZFlzRQ
This sounds like a dumb idea, and it probably is. But consider a few things:
* NVIDIA GPUs have exceptional memory bandwidth, and memory can be a slow resource on CPU based systems (perhaps limited by latency more than bandwidth)
* The clock speed isn't that slow, it's in the GHz. Still one's clocks per emulated instruction may not be great.
* You can still do pipelining, maybe enough to get the clocks-per-instruction down.
* Branch prediction can be done with ample resources. RNN based predictors are a shoe-in.
* communication between "cores" should be fast
* a many-core emulated CPU might not do too bad for some workloads.
* It would have good SIMD support.
Food for thought.