News for hackers. Stuff that matters.

A comment on Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference. Parent comment

Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by aurareturn on Tuesday November 19, 2024 @06:44AM (#42182387)
Each Cerebras wafer scale chip has 44GB of SRAM. You need 972 GB of memory to run Llama 405b at fp16. So you need 22 of these.

I assume they're using SRAM only to achieve this speed and not HBM.

[ Parent | on HN ]

There are no comments on this comment yet.