The writing has been on the wall for a while when it comes the next major bottleneck. I'm surprised that Intel was able to keep their quad-core game running for as long as they did, but "chiplets" were obviously going to have to happen, which stresses IO. IBM solved that a long time ago - as demonstrated by the massive IO bandwidth on display for several hardware generations. POWER9 came in two flavors, one with on chip memory controllers and one with their traditional off die strategy. So definitely not new - but its always been very boutique and therefor very expensive. Their OMI proposal was supposed to change that - but it won't, given the awful way they've executed it. Anybody remember that time that IBM's PS/2 offering had a major positive impact on the industry and didn't lead to a mess of dead ends? Me neither... this will be like that.

It goes both ways.

HBM on the die is crazy high performance.

With a bus like CXL you could have a card full of memory chips that is shared by multiple CPUs. The z machines have a system like this it is just proprietary for them.

Memory that is less coupled to the CPU won't be as fast as HBM but you can have a lot more of it.