News for hackers. Stuff that matters.

A comment on Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference.

Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by germanjoey on Monday November 18, 2024 @10:07PM (#42179744)
Pretty amazing speed, especially considering this is bf16. But how many racks is this using? The used 4 racks for 70B, so this, what, at least 24? A whole data center for one model?!
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by aurareturn on Tuesday November 19, 2024 @06:44AM (#42182387)
Each Cerebras wafer scale chip has 44GB of SRAM. You need 972 GB of memory to run Llama 405b at fp16. So you need 22 of these.

I assume they're using SRAM only to achieve this speed and not HBM.

[ Parent | on HN ]