News for hackers. Stuff that matters.

A comment on Groq runs Mixtral 8x7B-32k with 500 T/s. Parent comment

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by treesciencebot on Monday February 19, 2024 @12:49PM (#39432578)
per-chip compute is not the main thing this chip innovates for fast inference, it is the extremely fast memory bandwith. when you do that, you'll loose all of that and will be much worse off than any off the shelf accelerators.
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by QuadmasterXLII on Monday February 19, 2024 @03:46PM (#39434606)
load model, compute a 1k token response (ie, do a thousand forward passes in sequence, one per token), load a different model, compute a response,

I would expect the model loading to take basically zero percent of the time in the above workflow

[ Parent | on HN ]