News for hackers. Stuff that matters.

snhbsqub

Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B on Wednesday June 12, 2024 @11:35AM (#40659273)
- Groq has exactly 0 dollars in revenue - Groq requires 576 chips to run a single model - Groq can do low latency inference, but can't handle batches, and can't run a diversity of different models on each deployment - Groq quantizes the models, significantly affecting quality to get more speed (and don't communicate this to end users, which is very deceptive) - Groq can only run inference, cannot train on their systems

- SambaNova has real revenue from big customers - SambaNova can run any model on a single node at the speed Groq requires - SambaNova can do low latency inference just like Groq, but can also run large batches and host hundreds of models on a single deployment - SambaNova does not quantize models unless explicitly stated - SambaNova can run training at perf competitive with Nvidia, as well as fastest inference in the world at full precision

It really isn't a competition. Groq has done great as garnering hype in recent months, but it is a house of cards.

Re:SambaNova chip for LLMs nearly as fast as Groq on Saturday June 08, 2024 @01:35AM (#40615509)
Groq quantize their models, killing accuracy. SambaNova run the models at full precision, and on a single node, whilst Groq runs a single model on hundreds of chips. SambaNova run training and inference, Groq only run inference. SambaNova run hundreds on models on a single node without any performance degradation, Groq need hundreds of chips for a single model.

Groq are trying to launch their own cloud and running it at a loss to try to acquire users are raise venture capital so they can tape out a new chip (their current chip just is not very competitive, they get speed for having a good interconnect, but it is so cost ineffective). SambaNova sell systems to big companies and service providers, and don’t have a paid cloud API, so aren’t as visible as Groq. But SambaNova make real revenue whilst Groq have literally zero revenue, which is why they struggle to raise money.

Very soon people will realize that Groq are frauds and SambaNova is the legit challenger to Nvidia.

Re:SambaNova chip for LLMs nearly as fast as Groq on Saturday June 08, 2024 @01:31AM (#40615493)
Let me know how that goes.
Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B on Saturday June 08, 2024 @01:27AM (#40615472)
SambaNova is the only one if the chip startups that is viable. It surprises me that people don’t see this.
Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B on Saturday June 08, 2024 @01:24AM (#40615465)
Groq makes zero revenue, needs hundreds on chips to run 1 model, and runs everything at lower precision. SambaNova has a lot of revenue, and runs at that speed at full precision on a single node. It really isn’t a competition.
Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B on Saturday June 08, 2024 @01:22AM (#40615459)
How are they lagging? They are running faster than anyone else at full precision and with many many fewer chips than Groq. Groq is not real.