Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Groq Raises $640M to Meet Soaring Demand for Fast AI Inference. Parent comment
What models does Groq run at 500tok/sec? In what mode (e.g. batch size)?
UPD. found the model, it is Mixtral 8x7B-32k. AFAIK 8xH100 will do 100+tok/sec with batch size 1.
But that does not look too impressive for batch sizes higher than 1: https://www.baseten.co/blog/faster-mixtral-inference-with-te...
There are no comments on this comment yet.
What models does Groq run at 500tok/sec? In what mode (e.g. batch size)?
UPD. found the model, it is Mixtral 8x7B-32k. AFAIK 8xH100 will do 100+tok/sec with batch size 1.
But that does not look too impressive for batch sizes higher than 1: https://www.baseten.co/blog/faster-mixtral-inference-with-te...