Look at the Epiphany CPUs used as the co-processors in the Parallella [1] and you find something similar to what you suggest. The current iteration are 16 core CPU's in a fingernail sized package. They have a design for a 64 core CPU and says it will scale to 4096 core's on a single die.

It's fun to play with. The challenges are pretty much what people have been saying: To get to these core counts in the little space they have had to sacrifice cache size and memory size and single thread speed, even with a very clean and simple instruction set, and we are in general not good about taking advantage of high core counts other than in the GPU sense.

The Epiphany in variations with enough cores has the potential to beat GPUs for workloads with many independent instruction streams, but it'd gets crushed by GPUs of similar size for workloads that can be easily vectorised, and would crushed by any modern CPU for workloads that require high single core performance because of data dependencies, no matter the core count, exactly because most of the complexity it sacrifices cost single core performance.

The problem is that we don't really know how much space that leaves (the Parallella was designed largely as a development platform to let people experiment with an Epiphany CPU). Adapteva has focused on the low power usage of their CPU, and that may very well be a good idea.

[1] https://www.parallella.org/

GPUs have both pipelining and caches, so those don't serve to distinguish your idea from a GPU. Also remember that the barrier to spreading out computation over more cores is that many of our algorithms don't parallelize well, not that we can't fit 1024 cores into a computer. There's already been a huge incentive to get programs (even those used on regular desktops and not high performance computers) to parallelize well since we hit the clock-rate ceiling, but a lot of problems still have trouble saturating even a 4 core system.

What you are looking for is called the Mill:

https://millcomputing.com/

I don't think that is quite what he's looking for, the Mill is not exactly RISC.

Though it is an awesome architecture and I recommend anyone to check it out.

I mean, it's a cleanroom design unconstrained by backwards compatibility, and aspects of its design can be described as elegantly simple. You're right that it doesn't try to be RISC though.

That's literally what a GPU is.

That's literally not true.

GPUs have only something like 1-32 independent very wide MIMD/SIMD units with a lot of hardware threads.

Not sure about Nvidia 1080, but at least 980 has 32 SMM units, each roughly as defined above.

1024 independent cores... that'd be crazy. Surely they would not be cache coherent but only able to talk to immediate neighbors.

It'd be understatement to say that programming that kind of CPU would be challenging...