Re:Groqchat
by pclmulqdq on Saturday December 23, 2023 @12:12AM
(#38741809)
They are putting the whole LLM into SRAM across multiple computing chips, IIRC. That is a very expensive way to go about serving a model, but should give pretty great speed at low batch size.
[ Parent
| on HN ]
There are no comments on this comment yet.