In short, yes, higher capacity will involve higher latency.

The thing here is balancing core speed vs “effective” memory latency (factoring in cache hits, misses, etc).

Cache management is a hard problem, and the equilibrium point is load dependent (i.e. depends on what type of program you use).

AMD has been smart enough to understand that sometimes it’s better to just brute force your way in (with higher cache sizes) than being super clever about how you handle it.

At 192MB, just give me explicit access to this cache as memory and I’ll decide what goes into it. No need to try to predict what my code will want, I can just tell you.

Put everything in contiguous memory and load it, it'll be in cache

That not a great solution. My working data set is likely larger than 192MB. But my application is much more likely to know what data is in the hot path than the CPU is to guess it. I might want to put my DB index into this cache, or I might want to put only a part of it. I might want to preload the data then work on it for a bit, then load the next chunk. I might want one core preparing the data in RAM while another core is getting ready to work on it in this cache. Essentially I want full access to it independent of the main RAM because I can do a lot more with that. Think of this as the difference between RAM and disk: is RAM only useful as a cache to what’s on disk?

We know both positions are correct in practice. Most automatic prediction/management algorithms perform better on 90% of the code that’s run on them. However, careful performance tuning over time by experts can outperform generic solutions, sometimes dramatically.

I generally err on the side of “I’m not as smart as I think I am” in these kinds of discussions. It’s not that I can never do it. I’m sure if I studied it a bit I could. The reason is that 99% of the lines of code I’ve ever written don’t warrant that kind of attention. I’m sure there are uses like when you’re writing some core fundamental algorithm in the fields of compression, crypto algorithm, hash, video processing, etc etc etc. I don’t work in those spaces though so the benefit is much more marginal.

It’s possible 192mb is enough to start needing some explicit memory management. It makes the coding model much more complex though. And in a greedy software system where your code isn’t the only one running, such complexity doesn’t necessarily net overall wins. It’s the reason we have drivers and OSes even though we started with each piece of content bundling explicit HW support (at much better perf generally)

Imagine coding for a 64-bit version of the 6502, where you have a 192MB "zero page" :)

If you boot on actual hardware (no VM), if your application code + kernel (or unikernel) fits in under 192 megabytes, you are golden.

Actually, this is where 3d stacking changes the game. With 1 additional vertical layer, you can get double the memory at practically the same latency, as the source of latency is related to the wire distance (and capacitance) in the 2d plane. Latency can be lowered with 3d stacking as well if you reduce the 2d area of the memory array. This is how the industry will keep scaling going when 2d feature sizes can no longer be shrunk.

What's the limitation on adding additional vertical layers though? One apparent limitation could be that by adding several vertical layers it would require the wire to be longer.

Heat dissipation, although the power consumption of SRAM is mostly affected by I/O rather than capacity.

2 limitations.

1. Heat doesn't go away with layers and gets harder to cool. 2. These are still layers. You want each layer to be as flat as possible. As the layers deform you lose the ability to cleanly add more layers.

Current lithography is very much based on the notion that everything is flat.

AMD's SenseMI is a neural network to predict data fetching. That's quite the opposite of brute force.

They add massive caches as a side effect of having loads of cores enabled by their chipset designs.

Depends on your definition of “brute force”, I guess. Between explicitly deriving some algorithm from an understanding of causal relationships, v.s. training a neural net on billions of samples without encoding any broader understanding of the world, the latter approach is the one I would have called “brute force”.

Using NN for cache management is not that new, and is a simpler approach in comparison to Intels as far as I know.

Whilst it would be nice, we would be adding a lot more heat into the CPU block. Maybe they could and add extra cooling on the other side, though would really be looking at reinventing the CPU socket and thoughts of Intel Slot-1 form factor start to become appealing again as a form factor to allow such cooling solutions.

Now what we have over the years seen interest in is adding processing cores to the RAM itself and maybe having a small dedicated processor for some tasks attached to the RAM may well prove viable.

Imagine if we didn't have any CPU or RAM sockets/slots and just a row of slots you addedd a module that had CPU/RAM in one and can just add more upto the slot limit. But then, that is kinda how GPU's have gone already in many aspects and look at how much RAM they hold and how large the cooling solution for them is. That gives you an idea of the cooling needed for large amounts of processing and RAM when closely packaged.

> Now what we have over the years seen interest in is adding processing cores to the RAM itself and maybe having a small dedicated processor for some tasks attached to the RAM may well prove viable.

I think we call these 'GPUs' today.

> just a row of slots you addedd a module that had CPU/RAM

As you mentioned, GPUs fit this description, but some PCIe devices are full-blown embedded systems.

I think we're at the point where the only major improvements will take place on the CPU die itself.

GPUs aren't quite the same, because they're (mostly) SIMD. GPUs don't have the independent per-compute-core memory bandwidth required for every core to be running its own execution thread with an independent instruction pointer, independent data fetches, and (especially) non-correlated branch-prediction failures / cache misses. (This is most of the reason that branching, if even implemented in the GPU's ISA, is effectively useless.)

Whereas, what the parent poster is describing would be true MIMD: a bunch of tiny cores each with its own on-board RAM, its own instruction pointer, and then a bus (or a bunch of busses) fast enough to feed them all data from (probably NUMA) main memory.

GPUs don't provide any advantage for running e.g. a high-concurrency Erlang application server. But a true-MIMD system would.

I'm confident we'll see memory-on-chip within the next few years. Actually isn't that already the case for mobile CPU's?

Main memory, if you mean DRAM, no.

DRAM is incompatible with the logic family used in CPU making, though attempts were numerous at trying to work around that. Every few years, there are somebody coming with claims of passable CMOS+DRAM tech, but none got adoption so far.

I work on ferroelectric hafnium, which looks promising with regards to CMOS compatibility. It's non-volatile, though I guess you could also use it like DRAM. Endurance is an issue for now though (it's approaching RAM use-cases, but not SRAM yet).

As you said, I'm not sure about the density part. Unless it's completely done in BEOL, I don't see designers trade precious chip real estate for memory (unless pin or power-limited, of course).

Hafnium? Is it something FRAM related?

What do you think on FRAM vs. next generation MRAM?

I don't know much about STT-MRAM, so I'll avoid making a fool of myself. Looking more into the details is on my TODO. Look at the datasheets and make your choice, as usual. I am not sure how the situation will evolve on either side, my understanding is that FRAM could be more energy-efficient.

Well, hafnium can be used to make ferroelectric crystals, that are necessary for ferroelectric memories (FRAM). The most used ferroelectric material is PZT, which has lead, and is a nightmare for CMOS compatibility due to contamination issues, and temperatures.

It's a bit light on the details, but maybe I should point you towards the video that explains our project: https://www.youtube.com/watch?v=M8tL-nN7G-A

The most exciting part might not be the performance (though it looks good), but the way ferroelectrics can be used for new circuits (variable treshold transistors thanks to FeFETs). Hafnium has been used in gate oxides for a few years now so it's quite compatible with CMOS.

It is not actually incompatible within the same die, IBM z15 uses eDRAM for the L1 caches even.

Not L1, but L2 and on.

A very curious case, and a very clever hackery with a custom SOI process. It's still very, very far from coming to mainstream chips.

It says L1 here

"Both it and all levels of cache in the main processor from level 1 use eDRAM, instead of the traditionally used SRAM."

https://en.m.wikipedia.org/wiki/IBM_z15_(microprocessor)

I got L1 being SRAM from here https://fuse.wikichip.org/news/3383/ibm-doubles-its-14nm-edr...

IBM's technical guide also says it's eDRAM

https://www.redbooks.ibm.com/redbooks/pdfs/sg248851.pdf

It's almost sad to see the x86s compared to the IBM mainframes...

What is the main hurdle? Why are these processes incompatible?

>https://www.tomshardware.com/news/innovative-type-of-memory-...

No word on latency though.

Genuinely curious, could you say why logic and dram on the same chip are incompatible?

For practical considerations, both DRAM, and CMOS need a lot of steps, and both of them are already dialled closed to thermal limits of materials.

So if you just first make CMOS, and leave empty space nearby protected by something to do DRAM later, it will be very hard to fit into the tiny remaining thermal budget past which CMOS devices will turn into schmoo.

Basically, at the low nanometer range,[a] if you want on-chip DRAM, it’d be a separate die (“chiplet”) connected by the substrate

[a]: what’s the upper limit? As in, when does it not work anymore? For example, would it work at 100+ nm?

I am not a specialist, but I think I can guess.

First, you need a wafer on which you can make a capacitor, which already means an SOI process, and SOI wafers.

The lowest nodes with SOI available to mortals are 40nm, and 14nm on GlobalFoundries, but god knows how one gets GloFo to collaborate on that.

Then, you need CMOS device which has enough thermal budget to survive both own, and DRAM creation.

Third, I think on-die DRAM will only make sense when it wins over SRAM. DRAM cells cannot physically go smaller than the minimal size of a trench capacitor. I believe at 5-7nm nodes, 6T SRAM will already be smaller than a reasonably fast eDRAM per area.

We know that IBM's Z15 is a 14nm FinFet chip, and it has DRAM on board, with them probably somehow doing DRAM first.

That's because high-density logic and high-density DRAM require very different steps during wafer fabrication.

> I'm confident we'll see memory-on-chip within the next few years.

Depending on what you mean by "memory-on-chip" we've already seen this with AMD's Fiji (2015) & Vega (2017) line of GPUs which used on-package memory in the form of HBM/HBM2 memory ( https://www.anandtech.com/show/9390/the-amd-radeon-r9-fury-x... and https://www.anandtech.com/show/11717/the-amd-radeon-rx-vega-... ).

It was incredibly fast, but also very expensive and limiting in SKU configurations. The resulting 16GB SKU for the Vega 56/64, for example, made basically nobody happy. It was too much for gamers, who then didn't want to pay money for something that didn't help, and it wasn't enough for the professional crowd, who were getting used to 24GB offerings from Nvidia.

> Actually isn't that already the case for mobile CPU's?

Nope. They "just" replace the slot with solder more or less. It's still externally packaged DRAM modules.

Why are you confident about this? Semiconductor processes are complicated for a ton of reasons, and a process which is optimized for making main memory (DRAM) has some differences from a process optimized for making logic (CPU), and flash memory is different still.

I see AMD's effort as a way to sidestep the whole problem of putting memory and CPU on the same chip, by making separate chips and stacking them. It makes sense. You can get CPU packages with CPU + DRAM + Flash, but these are separate chips which are wired together inside the package.

E.g.

https://en.wikipedia.org/wiki/Package_on_a_package

The whole point of these systems is to avoid putting memory and CPU (and flash) on the same chip. AMD's version is smaller & more integrated than package-on-package but it still achieves the same goal: multiple chips.

You can find stacked chips pretty commonly, but it's effectively two chips within a single package.

Actually already a thing but production of the memory isn't easy so not as cost effective yet. HBM is used on the VEGA graphics. GPUs are a type of processor.

Mobile SoCs like A14 have a Package-on-Package architecture with DRAM die stacked on top of CPU, but it is not the same die.

As I imagine it, I immediately wonder how to cool it.

RAM typically doesn't generate much heat.

But IIUC you would at least need to pass the heat of the CPU though the RAM. So if you want to keep a lot of memory near the CPU you are at the very least adding some sort of "blanket" between the heat source and the heat sink.

Alternatively, you could put the CPU core on top of RAM, closest to the heat sink.

You also don't want to run your CPU at 80°C, else the RAM will produce much more errors. (ECC is a must anyway, I suppose.)

You can definitely do that, but now your RAM is twice as far away from the CPU on average than if you put RAM on all sides. The best solution is probably some sort of hybrid where you put RAM on all side, but more on the bottom to balance the thermal effects and the latency.

Thermal resistance is linear wrt the thickness of the material, so if the layer is sufficiently thin, it doesn't form a significant obstacle.

They also mentioned both dies are thinner so they are mechanically compatible with current heat spreaders.

On the other hand, 128MB is enough for my entire desktop environment and most of the (non-game) apps I use (obviously not at the same time.)