News for hackers. Stuff that matters.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by nilayj on Tuesday February 20, 2024 @07:06AM (#39440274)
How is the Token/second calculated? I ask it a simple prompt and the model generated a 150 word (about 300 tokens?) answer in 17 seconds, then mentioning the speed of 408T/s.

Also, I guess this demo would feel real time if you could stream the outputs to the UI? Can this be done in your current setup?

There are no comments on this comment yet.