Re:Kaist develops next-generation ultra-low power LLM accele...
by dloss on Thursday March 07, 2024 @04:02PM
(#39634536)
Which of the model requirements of Coral TPUs [1] are the most problematic for LLMs?
[1] https://coral.ai/docs/edgetpu/models-intro/#model-requiremen...
Re:Kaist develops next-generation ultra-low power LLM accele...
by semisight on Thursday March 07, 2024 @04:38PM
(#39634870)
Guessing as to what the GP meant--coral TPUs max out around 8M parameters, IIRC. That's a few orders of magnitude less than the smallest LLM model.
Re:Kaist develops next-generation ultra-low power LLM accele...
by moffkalast on Thursday March 07, 2024 @06:27PM
(#39635903)
The part where they have like 3 bytes of memory so you switch from extremely high latency of RAM to laughably sluggish latency of USB serial. I think there's also no support below 8 bit quants, which you'd really need.
>processes GPT-2 with an ultra-low power consumption of 400 milliwatts and a high speed of 0.4 seconds
Not sure what's the point on comparing the two, an A100 will get you a lot more speed than 2.5 tokens/sec. GPT 2 is just a 1.5B param model, a Pi 4 would get you more tokens per second with just CPU inference.
Still, I'm sure there's improvements to be made and the direction is fantastic to see, especially after Coral TPUs have proven completely useless for LLM and whisper acceleration. Hopefully it ends up as something vaguely affordable.