News for hackers. Stuff that matters.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @02:56PM (#39433990)
We are not using 500 GPUs, we are using a large system built from many of our own custom ASICs. This allows us to do batch size 1 with no reduction in overall throughput. (We are doing pipelining though, so many users are using the same system at once).

There are no comments on this comment yet.