Groq runs Mixtral 8x7B-32k with 500 T/s

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by ppsreejith on Monday February 19, 2024 @08:41AM (#39429609)

Thanks again! Hope I'm not overwhelming but one more question: Are you decoding with batch size = 1 or is it more?

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @08:47AM (#39429663)

That's OK, feel free to keep asking!

I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.

[ Parent | on HN ]