News for hackers. Stuff that matters.
Re:SambaNova chip for LLMs nearly as fast as Groq by coder1001 on Wednesday May 29, 2024 @01:39PM (#40514628)
Both are faster than NVIDIA! Always good to see competition!
Re:SambaNova chip for LLMs nearly as fast as Groq by anon291 on Wednesday May 29, 2024 @04:01PM (#40516317)
I mean that 'almost as fast as Groq' is not a compelling strategy. If it's almost as fast, why not just go with Groq? Are they going for cheaper? Added services? I'm just wondering.
Re:SambaNova chip for LLMs nearly as fast as Groq by vessenes on Thursday May 30, 2024 @03:07PM (#40527477)
Well, Groq memory-per-rack looks very low. That is to say, the whole world now understands how exciting very high inference throughput is in a way that almost nobody did when Groq started, (I think I saw an early pitch deck, and don't recall fast inference as even a differentiator in that deck, although I could be wrong). However, the number of Groq chips/servers/whatever you call them that are needed to get up to running Llama 400B looks like a lot. Like many, many racks worth.

Plus, Groq claims to have converted over to only being a cloud provider now, and will keep their hardware to themself. Given that I can't even sign up for a pay-as-you-go API key with Groq right now, I think there's a lot of room for competitors.

Re:SambaNova chip for LLMs nearly as fast as Groq by snhbsqub on Saturday June 08, 2024 @01:35AM (#40615509)
Groq quantize their models, killing accuracy. SambaNova run the models at full precision, and on a single node, whilst Groq runs a single model on hundreds of chips. SambaNova run training and inference, Groq only run inference. SambaNova run hundreds on models on a single node without any performance degradation, Groq need hundreds of chips for a single model.

Groq are trying to launch their own cloud and running it at a loss to try to acquire users are raise venture capital so they can tape out a new chip (their current chip just is not very competitive, they get speed for having a good interconnect, but it is so cost ineffective). SambaNova sell systems to big companies and service providers, and don’t have a paid cloud API, so aren’t as visible as Groq. But SambaNova make real revenue whilst Groq have literally zero revenue, which is why they struggle to raise money.

Very soon people will realize that Groq are frauds and SambaNova is the legit challenger to Nvidia.