News for hackers. Stuff that matters.
Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B by andy_xor_andrew on Thursday May 30, 2024 @03:14PM (#40527552)
When reading Hacker News you develop a signal/noise filter, where lots of headlines make bold claims but you filter them out as embellishment or exaggeration.

My bullshit detector went off when I first saw Groq posted on HN - a startup is making their own chips (doubt) that performs faster than anything Nvidia has for inference (doubt) and accelerates LLMs to hundreds/thousands of tokens per second?? Mega doubt.

But... then I tried their demo, and... yeah, it's that good. Such an amazing company of talented individuals.

Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B by saberience on Thursday May 30, 2024 @03:24PM (#40527662)
The issue is that their chips need a huge amount of server blades and there's a big doubt whether this model actually scales. That is, how will Groq handle much larger models with a context of hundreds of thousands or millions of tokens? Right now this would require them to deploy a cluster with thousands of chips, versus 10 chips for say an NVidia system.

The other issue they don't mention is power, space, efficiency etc. We want to run larger models with less power, fewer server blades, at lower cost. Not use more server blades, more chips, more power, etc.

Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B by verdverm on Thursday May 30, 2024 @04:13PM (#40528074)
Cerebrus faces similar challenges with their wafer scale chips.

If anything, Google's TPU advancements chart a viable course. I suspect both Groq and Cerebrus will overcome the challenges and offer competitive compute options, depending on the context

Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B by snhbsqub on Saturday June 08, 2024 @01:27AM (#40615472)
SambaNova is the only one if the chip startups that is viable. It surprises me that people don’t see this.
Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B by frozenport on Thursday May 30, 2024 @11:06PM (#40531104)
8 year old unicorn++ with a public demo sounds credible?