Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by tome on Monday February 19, 2024 @08:47AM
(#39429663)
That's OK, feel free to keep asking!
I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.
I think currently 1. Unlike with graphics processors, which really need data parallelism to get good throughput, our LPU architecture allows us to deliver good throughput even at batch size 1.