Re:Groq surpasses 1,200 tokens/sec with Llama 3 8B
by verdverm on Thursday May 30, 2024 @04:13PM
(#40528074)
Cerebrus faces similar challenges with their wafer scale chips.
If anything, Google's TPU advancements chart a viable course. I suspect both Groq and Cerebrus will overcome the challenges and offer competitive compute options, depending on the context
The other issue they don't mention is power, space, efficiency etc. We want to run larger models with less power, fewer server blades, at lower cost. Not use more server blades, more chips, more power, etc.