News for hackers. Stuff that matters.

A comment on Groq runs Mixtral 8x7B-32k with 500 T/s. Parent comment

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by WiSaGaN on Monday February 19, 2024 @10:33AM (#39430863)
How much do 568 chips cost? What’s the cost ratio of it comparing to setup with roughly the same throughput using A100?
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by benchess on Monday February 19, 2024 @10:42AM (#39430971)
They’re for sale on Mouser for $20625 each https://www.mouser.com/ProductDetail/BittWare/RS-GQ-GC1-0109...

At that price 568 chips would be $11.7M

[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @10:46AM (#39431004)
Yeah, I don't know what the cost to us is to build out our own hardware but it's significantly less expensive than retail.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by WiSaGaN on Monday February 19, 2024 @10:47AM (#39431020)
That seems to be per card instead of chip. I would expect it has multiple chips on a single card.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by renewiltord on Monday February 19, 2024 @10:52AM (#39431099)
From the description that doesn't seem to be the case, but I don't know this product well

> Accelerator Cards GroqCard low latency AI/ML Inference PCIe accelerator card with single GroqChip

[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by WiSaGaN on Monday February 19, 2024 @10:54AM (#39431121)
Missed that! Thanks for pointing out!
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by fennecbutt on Monday February 19, 2024 @09:06PM (#39437226)
I presume that's because it's a custom asic not yet in mass production?

If they can get costs down and put more dies into each card then it'll be business/consumer friendly.

Let's see if they can scale production.

Also, where tf is the next coral chip, alphabet been slacking hard.

[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by bethekind on Tuesday February 20, 2024 @12:18AM (#39438297)
I think Coral has been taken to the wooden shed out back. Nothing new out of them for years sadly
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by fennecbutt on Tuesday February 20, 2024 @06:33AM (#39440083)
Yeah. And it's a real shame bc even before LLMs got big I was thinking, couple generations down the line and coral would be great for some home automation/edge AI stuff.

Fortunately LLMs and hard work of clever peeps running em on commodity hardware are starting to make this possible anyway.

Because Google Home/Assistant just seems to keep getting dumber and dumber...

[ Parent | on HN ]