Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by qeternity on Monday February 19, 2024 @07:17PM
(#39436490)
It’s important…if you’re building a chatbot.
The most interesting applications of LLMs are not chatbots.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by chasd00 on Monday February 19, 2024 @10:14PM
(#39437652)
> The most interesting applications of LLMs are not chatbots.
What are they then? Every use case I’ve seen is either a chatbot or like a copy editor which is just a long form chatbot.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by jasonjmcghee on Monday February 19, 2024 @11:22PM
(#39438037)
Obviously not op, but these days LLMs can be fuzzy functions with reliably structured output, and are multi-modal.
Think about the implications of that. I bet you can come up with some pretty cool use cases that don't involve you talking to something over chat.
One example:
I think we'll be seeing a lot of "general detectors" soon. Without training or predefined categories, get pinged when (whatever you specify) happens. Whether it's a security camera, web search, event data, etc
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by nycdatasci on Monday February 19, 2024 @11:28PM
(#39438069)
Complex data tagging/enrichment tasks.
Re:Groq runs Mixtral 8x7B-32k with 500 T/s
by throwaway2037 on Monday February 19, 2024 @10:41PM
(#39437801)
> The most interesting applications of LLMs are not chatbots.
In your opinion, what are the most interesting?
I believe that this is doable - my pipeline is generally closer to 400ms without RAG and with Mixtral, with a lot of non-ML hacks to get there. It would also definitely be doable with a joint speech-language model that removes the transcription step.
For these use cases, time to first byte is the most important metric, not total throughput.