You need think of it as aggregate memory, not as per core memory to use if effectively. Are you aware of a chip with more than 64MB of on chip RAM?

The latest generations of IBM Power processors have >64MB L3 caches on chip. The Power 7+ has 80MB per chip, the 12 core Power 8 96MB, according to Wikipedia the Power 9 will have 120MB.

Good data! That puts e5 in good company with some big-iron heavies.

I realize comparing to Intel is unfair but I think the Skylake Iris Pro 580 has 128MB on chip RAM. https://en.wikipedia.org/wiki/Intel_HD_and_Iris_Graphics#Sky...

That's eDRAM though, and it's really on-module rather than on-chip. It's a separate die on the same module as the main SoC.

64 MB static RAM, no less. You've built a huge-ass static RAM chip and thrown in some local processing. (-:

Consider that many instruction and data caches are at the 16-32 KB scale. It's obviously a big criticism of the microarchitecture but you have a linear tradeoff between number of cores and available core memory. One core with 64 MB of memory seems less useful than 1024 cores with 64 KB of memory each (which can directly access all other core memory). But 65,536 cores with 1KB of memory each doesn't sound very useful either.

Thanks for articulating. As you know, there is no right answer as it depends on workload. Now if we could only build a specific chip for every application domain....

In fact, you have two trade-offs. One is what you said - that for a fixed amount of memory, the more cores, the less memory you have per core. The second trade-off is the transistor budget - the more space you use for cores, the less space you have left for memory.

The third trade off is cycle time; the larger the memory, the longer it takes to access it. This is why L1 caches are typically 16-64 KiB and despite that access is typically 2-3 cycles. However, 3+ cycles is difficult to hide in an in-order processor like this.

> But 65,536 cores with 1KB of memory each doesn't sound very useful either.

You've just described the general architecture of the Connection Machine[0], a late 80's early 90's era supercomputer that was used for modeling weather, stocks, and other items. It was fairly useful in it's time.

[0]https://en.wikipedia.org/wiki/Connection_Machine

I think the right way to think about this is the following: scaling "up" is basically over with CPUs. Now we need scaling "out". This means learning how to make use of many more smaller cores, rather than just a few larger ones. Here communications becomes the problem, and indirectly, affects how you design and implement software. Scaling is becoming a software problem: how can you take advantage of 1024 cores with just 64KB or memory each, in a world where terabyte-sized is the daily business?

I think we will end up with systems with 64GB of memory, but which instead of 8 cores with 8GB each, have 1M cores with 64 KB memory each. We just need to lean how to write code that makes the most out of that, which is probably a lot more than what you can do with current systems.

And this Epiphany thing is something like the first step in that direction.

Exciting times.

PS3 SPUs had 256KB, you'll want to vectorize your data anyway if you want to take of advantage of this.

You can't always vectorize your data. Like if you want to do highly parallel 3D rendering, you need the whole scene accessible to each core.

Of course, in that situation, the scene probably fits in 64 MB, so it's not really a limitation.

Unfortunately not, at least not for the real work type of scenes that you see in movies / cartoons. Textures and high-polygon models take a ton of space.

Depends. If you do 3D rendering with triangles and shaders you can divide your buffers into tiles based on storage size and stream vertex/shader commands.

This is actually how all modern mobile GPUs work and it's highly vectorizable. The partitioning obviously needs to know the whole scene but that's much more lightweight than rendering.

From what I've heard from my ex-gamedev contacts movies are heading that route in a large way because the turnaround time a raytracing is so long that's it's really hurting the creative process.

The current top500 machine has 64KB scratchpad per processing core and seems to be capable of running real HPC applications well <http://www.netlib.org/utk/people/JackDongarra/PAPERS/sunway-....

Depending on the application this may be a reasonable trade-off.