Groq runs Mixtral 8x7B-32k with 500 T/s

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @08:37AM (#39429575)

You're welcome! Yes, we have KV cache. Being able to implement this efficiently in terms of hardware requirements and compute time is one of the benefits of our deterministic chip architecture (and deterministic system architecture).

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by ppsreejith on Monday February 19, 2024 @08:41AM (#39429609)

Thanks again! Hope I'm not overwhelming but one more question: Are you decoding with batch size = 1 or is it more?

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @08:47AM (#39429663)

That's OK, feel free to keep asking!

I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.

[ Parent | on HN ]