That said, just because they can't use it right off the bat doesn't mean they won't figure it out over time. We really didn't get to see them try. The Xbox used a non-power-of-2, lame core count while Cell didn't use traditional cores at all. Apple to oranges a bit although 360 is a realistic comparison if the multicore tooling sucks.
So, what's Microsoft and Sony's recent approach? Drop a bunch of cores on them. :)
The problem with Cell was not that it had multiple cores. It was that it had to do a lot of graphics tasks to take the load off the weak GPU. And doing graphics on a CPU is quite hard. Doing so in concert with another GPU is doubly so. On the other hand, doing CPU work on GPU is much easier.
You can think of GPU cores as you schedule a series of "vector" instruction over a large chunk of data. They are great at internally paralleling this process because the flow is simple. You assume the there will be little synchronization and little branching. That's why you need such fast memory on GPUs to keep streaming data at instruction speed.
CPU cores are a lot more general prepuce where you're generally building the business logic and the work load is a lot more branch oriented.
So if you have a problem that's trivially parallizable without (much) branching logic then throwing more GPU cores is easy. Problems like this numerical in nature like large matrix calculations (math or 3d meshes), streaming data (hashing).
Hence why it's easier to keep adding GPU cores... it means games can process more geometry data in a fame, but adding more general prepuce CPU cores is harder because the complexity of business logic goes up as you split up the work and have to worry about ordering/synchronization.
Of course, you don't have to run the same code in every thread. If you can figure hundreds of different tasks to do simultaneously in a game - you can put each one of them in a thread too.
This is also why a GPUs, generally, use a very slow memory, contrary to your belief, With so many threads latency does not matter that much since a thread that gets stalled on a memory access is preempted for one that already has data available.
The current GPUs effectively have the fastest off core memory of current devices. They need those transfer rates to keep all the stream processors running.
Bandwidth is great though, but if bandwidth had been speed you could also say that a container ship is faster than a supersonic jet.
GPUs cards makers the trade of between bandwidth and latency i n favor of latency. When you're doing mostly branch free processing in large chunks that's the trade of to make. All you need is a strait forward pre-fetcher and you don't need to worry about latency.
That's not true for general purpose CPUs that perform lots of branches, that need to predicted (so we can predict what to fetch). The data processed on CPUs tends to be different (structures vs. vectors) and lots of pointer chasing (vtables, linked lists, hash tables, trees). That requires lower latencies since the access pasterns a lot more random.
The stated goal of HBM is taming the power consumption (and thus also heat) of GPU systems while keeping the same (or higher) bandwidth. The name HBM stands for high bandwidth memory.
And while HBM has a lower clock frequency compared to GDDR5 (like 1/4) it has a much wider bus. The bus on HBM is 1024 bits vs 32 bits for GDDR. At one time it can send 32x times the data in the bus. 32x / 4 = 8. The transfer rate of is 8 times bigger. The recent radeon cards with HBM now have memory transfer speeds of 256GB/s vs the 48GB for GGDR5.
Again, HBM trades latency for bandwidth. It negates some the latency drops due to 1/4 of the clock by putting the HBM memory on die vs off die.
I think you're conflating a few different arguments. GPU workloads are not latency sensitive, so in GPU land transfer speed (bandwidth) is speed.