Groq runs Mixtral 8x7B-32k with 500 T/s

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by Aeolun on Monday February 19, 2024 @07:37PM (#39436642)

I don’t really understand this. If you are happy to buy a <2K card, then what does it matter if the service is paid or not? Clearly you have enough disposable income to not care about a ‘free’ tier.

[ Parent | on HN ]

Re:Groq runs Mixtral 8x7B-32k with 500 T/s by frognumber on Tuesday February 20, 2024 @01:40PM (#39445060)

There's two questions. Why local?

1) Privacy and security. I work with PII.

2) Low-level access and doing things the manufacturer did not intend, rather than just running inference on Mixtral.

3) Knowing it will be there tomorrow, and I'm not tied to you. I'm more than happy to pay for hosted services, so long as I know after your next pivot, I'm not left hanging.

Why free tier?

I'm only willing to subsidize my employer on rare occasions.

Paying $12 for a prototype means approvals and paperwork if employer does it. I won't do it out-of-pocket unless I'm very sure I'll use it. I've had free tier translate into millions of dollars of income for one cloud vendor about a decade ago. Ironically, it never happened again, since when I switched jobs, my free tier was gone.

[ Parent | on HN ]