News for hackers. Stuff that matters.

A comment on Groq runs Mixtral 8x7B-32k with 500 T/s. Parent comment

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by phh on Monday February 19, 2024 @11:16AM (#39431431)
You're running fp32 models, fp16 or quantized?
[ Parent | on HN ]
Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @12:01PM (#39432025)
FP16 for calculating all activations. Some data is stored as FP8 at rest.
[ Parent | on HN ]