Groq runs Mixtral 8x7B-32k with 500 T/s

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by fennecbutt on Monday February 19, 2024 @09:06PM (#39437226)

I presume that's because it's a custom asic not yet in mass production?

If they can get costs down and put more dies into each card then it'll be business/consumer friendly.

Let's see if they can scale production.

Also, where tf is the next coral chip, alphabet been slacking hard.

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by bethekind on Tuesday February 20, 2024 @12:18AM (#39438297)

I think Coral has been taken to the wooden shed out back. Nothing new out of them for years sadly

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by fennecbutt on Tuesday February 20, 2024 @06:33AM (#39440083)

Yeah. And it's a real shame bc even before LLMs got big I was thinking, couple generations down the line and coral would be great for some home automation/edge AI stuff.

Fortunately LLMs and hard work of clever peeps running em on commodity hardware are starting to make this possible anyway.

Because Google Home/Assistant just seems to keep getting dumber and dumber...

[ Parent | on HN ]