News for hackers. Stuff that matters.

A comment on Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference. Parent comment

Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by danpalmer on Tuesday November 19, 2024 @12:39AM (#42180357)
In that case I'm misunderstanding you. Are you saying that it's "BS" that they are reaching ~1k tokens/s? If so, you may be misunderstanding what a Cerebras machine is. Also 8xH100 is still ~half the price of a single Cerebras machine, and that's even accounting for H100s being massively over priced. You've got easily twice the value in a Cerebras machine, they have nearly 1m cores on a single die.
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by sam_dam_gai on Tuesday November 19, 2024 @01:09AM (#42180469)
Ha ha. He probably means ”at a batch size of 1”, i.e. not even using some amortization tricks to get better numbers.
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by danpalmer on Tuesday November 19, 2024 @01:23AM (#42180534)
Ah! That does make more sense!
[ Parent | on HN ]