Groq runs Mixtral 8x7B-32k with 500 T/s

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by tome on Monday February 19, 2024 @03:30PM (#39434442)

Yup, understood. Access to consumer hardware like this is not something that we provide at the moment, I'm afraid.

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by frognumber on Monday February 19, 2024 @04:51PM (#39435269)

Don't blame you. Been at plenty of startups, resources are finite, and focus is important.

My only point was to, well, perhaps bump this up from #100 on your personal priority list perhaps to #87, to the limited extent that influences your business.

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by matanyal on Monday February 19, 2024 @06:05PM (#39435896)

Groq Engineer here as well; we actually built our compiler to compile pytorch, TensorFlow, and Onnx natively, so a lot of the amazing work being done by y'all isn't building much of a moat. We got LLama2 working on our hardware in just a couple of days!

[ Parent | on HN ]