I think the NOC approach is fundamentally similar to Intel's rings that they used for core interconnect back in the mid-2010s. It works.

https://www.realworldtech.com/includes/images/articles/snbep...

https://en.wikichip.org/wiki/intel/microarchitectures/sandy_...

They still use this today, and the ring interconnect is also the topology inside zen3/zen4 CCXs (with 8 cores). Ring is one of the simplest and best systems for >4 cores, until you get to about 8-10 cores (at which point you generally split it into multiple “tiers” like multiple CCXs etc).

Intel switched to a mesh model: https://en.wikichip.org/wiki/intel/mesh_interconnect_archite...

not for consumer chips, which still use the same ringbus design.

e-cores do have a CCX/core cluster, but the clusters themselves go on the ringbus lol

I think the distinction with ML inference workloads is that you often have very little control flow, so this type of architecture lets you match layers to adjacent cores so that each operation gets it's data directly from the step before rather than from RAM.

Right, so latency is less of an issue (because control flow is predictable) and performance becomes about bandwidth.