Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by kkzz99 on Monday February 19, 2024 @12:02PM
(#39432040)
How does the Groq PCIE Card work exactly? Does it use system ram to stream the model data to the card? How many T/s could one expect with e.g. 36000Mhz DDR4 Ram?
[ Parent
| on HN ]
We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by kkzz99 on Monday February 19, 2024 @12:31PM
(#39432340)
How does that work when the card only has 230MB of SRAM?
We connect hundreds of chips across several racks with fast interconnect.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by AhtiK on Monday February 19, 2024 @02:42PM
(#39433817)
How fast is the memory bandwidth of that fast interconnect?
Have a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public.
https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...