Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by nilayj on Tuesday February 20, 2024 @07:06AM
(#39440274)
How is the Token/second calculated? I ask it a simple prompt and the model generated a 150 word (about 300 tokens?) answer in 17 seconds, then mentioning the speed of 408T/s.
Also, I guess this demo would feel real time if you could stream the outputs to the UI? Can this be done in your current setup?
Also, I guess this demo would feel real time if you could stream the outputs to the UI? Can this be done in your current setup?