News for hackers. Stuff that matters.

A comment on Llama 3.1 405B now runs at 969 tokens/s on Cerebras Inference. Parent comment

Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by qeternity on Monday November 18, 2024 @11:56PM (#42180185)
I'm not talking about that. I and many others here have spun up 8x or more H100 clusters and run this exact model. Zero other traffic. You won't come anywhere close to this.
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by danpalmer on Tuesday November 19, 2024 @12:39AM (#42180357)
In that case I'm misunderstanding you. Are you saying that it's "BS" that they are reaching ~1k tokens/s? If so, you may be misunderstanding what a Cerebras machine is. Also 8xH100 is still ~half the price of a single Cerebras machine, and that's even accounting for H100s being massively over priced. You've got easily twice the value in a Cerebras machine, they have nearly 1m cores on a single die.
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by sam_dam_gai on Tuesday November 19, 2024 @01:09AM (#42180469)
Ha ha. He probably means ”at a batch size of 1”, i.e. not even using some amortization tricks to get better numbers.
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by danpalmer on Tuesday November 19, 2024 @01:23AM (#42180534)
Ah! That does make more sense!
[ Parent | on HN ]
Re:Llama 3.1 405B now runs at 969 tokens/s on Cerebras Infer... by aurareturn on Tuesday November 19, 2024 @02:13AM (#42180762) 

  I'm not talking about that. I and many others here have spun up 8x or more H100 clusters and run this exact model. Zero other traffic. You won't come anywhere close to this.
8x H100 can also do fine tuning right? Does Cerebras offer fine tuning support?
[ Parent | on HN ]