News for hackers. Stuff that matters.

A comment on Groq runs Mixtral 8x7B-32k with 500 T/s. Parent comment

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @12:23PM (#39432265)
We build out large systems where we stream in the model weights to the system once and then run multiple inferences on it. We don't really recommend streaming model weights repeatedly onto the chip because you'll lose the benefits of low latency.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by kkzz99 on Monday February 19, 2024 @12:31PM (#39432340)
How does that work when the card only has 230MB of SRAM?
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @12:34PM (#39432392)
We connect hundreds of chips across several racks with fast interconnect.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by AhtiK on Monday February 19, 2024 @02:42PM (#39433817)
How fast is the memory bandwidth of that fast interconnect?
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @02:53PM (#39433955)
Have a look at section 2.3 of our paper. Between any two chips we get 100 Gbps. The overall bandwidth depends on the connection topology used. I don't know if we make that public.

https://wow.groq.com/wp-content/uploads/2023/05/GroqISCAPape...

[ Parent | on HN ]