It's 40 GB of SRAM. I doubt it supports external memory.

> Also, if the chip is the size of a wafer, is it appropriate to call it a Chip?

Good question. I think it is. I mean the word "chip" isn't really that well defined (is HBM one chip?), but given that they sell it as a single unit and you can't really cut it in half I think it's one chip.

Looking at the picture, the thing is a platter. Not a chip.

It's a really cool picture too.

It's a wafer, but that doesn't preclude it from being a chip. Where's you definition of chip coming from.

Considering the chipped edges, it is not far off.

40GB of SRAM. Not quite big enough to fit the big models like GPT3

I don’t think it’s straightforward to do a head to head comparison.

from: https://www.youtube.com/watch?v=yso2S2Svdlg

@ 25:14

James Wang: "If a model doesn’t fit into a GPU’s HBM, is it smaller when it’s laid out in the Cerebras way relative to your 18 gigabytes?"

Andrew Feldman: "It is — it’s smaller in that we hold different things in memory than they do. One can imagine a model that has more parameters than we can hold — one can posit one, but remember our memory is doing different things. Our memory is basically holding parameters. That’s not what their memory is doing. Their memory is holding the shape of the model, their model is holding the results of the batches. We use memory rather differently. We haven’t found models that we can’t place and train on a chip. We expect them to emerge, that’s why we support clustering of chips and systems, that’s why we do that in whats called a “model parallel” way, where If you put two chips together you get twice the memory capacity. That’s not what you get when you put multiple GPUs together. When you put multiple GPUs together you get two versions of the same amount of memory, you actually don’t get twice the memory. I see you smiling here because you know that’s a problem… …With us if we support 4 billion parameters and you add a second wafer scale engine, now you support 8 billion parameters ,and if you add a third you can support 12 billion. That’s not the way it works with GPUs. With GPUs you just support two chips, each with a few million - tens of millions of parameters."

Are there any good resources out there describing in practice how existing training workloads are distributed among GPUs? (using tensorflow, pytorch, or whatever else?).

I'm curious how the problem effectively gets sliced.

SOTA on the biggest language models (which is where effectively the largest models are) is here: https://www.microsoft.com/en-us/research/blog/zero-infinity-...

It seems like you typically want a balance of memory size/bandwidth to compute ratio for typical deep learning applications.

The 40GB of SRAM probably has tremendous bandwidth (it could all be updated every few cycles!), but the memory size is very small compared to the amount of compute available.

However, maybe a different way of looking at it is that this chip will allow the training steps on deep learning models to take a fraction of the time as a GPU. Perhaps what takes 1s on a GPU could take 10ms on this chip.

So, this product may be effective at making training happen very fast, but without substantial model size or efficiency gains.

That's still ground breaking -- you can't acheive this result on GPUs. You can't achieve this result by any parallelization or distributed training, either. The large batch sizes in distributed training do not result in the same model or one that generalizes as well.

SRAM (static RAM) vs DRAM (dynamic RAM) for anybody else curious: https://computer.howstuffworks.com/question452.htm

Tom's Hardware has some nice tables comparing the specs: https://www.tomshardware.com/news/cerebras-wafer-scale-engin...

(more than the IEEE)

This thing pulls 15-20kW of juice!

> This thing pulls 15-20kW of juice!

If you look at the wafer she’s holding at the top, it’s seemingly segmented into a 12x7 grid of roughly chip-sized rectangles. That’s 84 “CPUs” at 200-240 watts each, which is pretty well in line with discrete server CPUs.

The amount of heat coming off this thing must be amazing, though.

I did a double that when I realised that was Kilowatts not Watts. This chip uses more energy in an hour than the average household (in my country at least) does in a day.

It may be a very large wafer but dissipating that heat is still very impressive.

It sounds like a lot but it almost isn't? Like this is ~50x bigger than an Nvidia A100, and the A100 pulls up to 400w. 50 * 400 ~= 20kW. So in terms of thermal density it's in-line with existing GPUs.

That said, I'd be fascinated to see the cooling solution. Is it just a massive copper heatsink & a boatload of airflow? Typical approaches of using heatpipes to expand the heatsink won't really work with something this big after all. Or is it a massive waterblock with multiple inlets/outlets so it can hit up a stack of radiators? How do they get even mounting pressure across that large of an area?

There's an image on their website[1], pretty huge water pumps.

"To solve the 70-year-old problem of wafer-scale, we needed not only to yield a big chip, but to invent new mechanisms for powering, packaging, and cooling it.

The traditional method of powering a chip from its edges creates too much dissipation at a large chip’s center. To prevent this, CS-2’s innovative design delivers power perpendicularly to each core.

To uniformly cool the entire wafer, pumps inside CS-2 move water across the back of the WSE-2, then into a heat exchanger where the internal water is cooled by either cold datacenter water or air."

[1]: https://cerebras.net/product/