News for hackers. Stuff that matters.

A comment on Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference. Parent comment

Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by sam_dam_gai on Tuesday November 19, 2024 @01:09AM (#42180469)
Ha ha. He probably means ”at a batch size of 1”, i.e. not even using some amortization tricks to get better numbers.
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by danpalmer on Tuesday November 19, 2024 @01:23AM (#42180534)
Ah! That does make more sense!
[ Parent | on HN ]