News for hackers. Stuff that matters.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by Voloskaya on Wednesday July 23, 2025 @12:58PM (#44661393)
> I really don't see the advantage long term over smaller chips like GB200 since both architectures use HBM.

I don’t think you can look at those things binarily. 44GB of SRAM is still a massive amount. You don’t need infinite SRAM to get better performances. There is a reason NVidia is increasing the L2 cache size with every generation rather than just sticking with 32MB if it really changed nothing to have a bit more. The more SRAM you have the more you are able to mask communication behind computation. You can imagine with 44GB being able to load the weights of layer N+1 into SRAM while computing layer N, thereby entirely negating the penalty of going to HBM (same idea as FSDP).

Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by vlovich123 on Wednesday July 23, 2025 @02:04PM (#44662156)
> You can imagine with 44GB being able to load the weights of layer N+1 into SRAM while computing layer N, thereby entirely negating the penalty of going to HBM (same idea as FSDP).

You would have to have an insanely fast bus to prevent I/O stalls with this. With a 235B fp16 model you’d be streaming 470GiB of data every graph execution. To do that 1000tok/s, you’d need a bus that can deliver a sustained ~500 TiB/s. Even if you do a 32 wide MoE model, that’s still about 15 TiB/s of bandwidth you’d need from the HBM to avoid stalls at 1000tok/s.

It would seem like this either isn’t fp16 or this is indeed likely running completely out of SRAM.

Of course Cerebas doesn’t use a dense representation so these memory numbers could be way off and maybe that is SRAM+DRAM combo