News for hackers. Stuff that matters.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by imtringued on Wednesday July 23, 2025 @09:12AM (#44658851)
Yeah sure, but if you do that you are heavily dropping the token/s for a single user. The only way to recover from that is continuous batching. This could still be interesting if the KV caches of all users fit in SRAM though.
Re:Cerebras Launches Qwen3-235B, Achieving 1,500 Tokens per ... by Voloskaya on Wednesday July 23, 2025 @09:26AM (#44659011)
> but if you do that you are heavily dropping the token/s for a single user.

I don’t follow what you are saying and what “that” is specifically. Assuming it’s referencing using HBM and not just SRAM, this is not optional on a GPU, SRAM is many order of magnitudes too small. Data is constantly flowing between HBM and SRAM by design, and to get data in/out of your GPU you have to go through HBM first, you can’t skip that.

And while it is quite massive on a Cerebras system it is also still too small for very large models.