News for hackers. Stuff that matters.
Re:OpenAssistant Conversations – Democratizing Large Languag... by bugglebeetle on Saturday April 15, 2023 @05:49PM (#35584902)
I tried a few prompts I use in production stuff and it failed on all of them and hallucinated quite a bit more. All of these models are optimized for the gimmicky chatbot stuff that seems impressive to a casual user, but not for comparable capabilities to GPT-3.5. I wish what the parent said was true because it would save me money!
Re:OpenAssistant Conversations – Democratizing Large Languag... by akiselev on Saturday April 15, 2023 @06:32PM (#35585290)
Which open model comes closest to GPT-3.5 in your production workload, if you don't mind me asking?
Re:OpenAssistant Conversations – Democratizing Large Languag... by bugglebeetle on Saturday April 15, 2023 @06:39PM (#35585354)
None of them really, because I use complex prompts with task breakdowns that no other models beside OpenAI’s seem capable of processing. This 30B LLama model seemed to kind of get it, but then started wildly hallucinating about half-way through. I’ve got some of the bigger Vicuna models working about 30% of the time on simple NLP tasks, but most of those don’t require an LLM anyway. They might perform better if you fine-tune them for whatever particular job, but that kind of defeats the purpose. The advantage of LLMs is supposed to be their generalized capabilities.
Re:OpenAssistant Conversations – Democratizing Large Languag... by xbmcuser on Saturday April 15, 2023 @11:18PM (#35586968)
I think most people don't realise that OpenAi biggest advantage is the billions of queries it was asked those signals are what they used to optimise it. So I think very hard for a local model to reach similar capability.
Re:OpenAssistant Conversations – Democratizing Large Languag... by pmontra on Sunday April 16, 2023 @04:07AM (#35588149)
I wonder how they use those queries for training. Maybe they use those responses that the user answered with great, thanks and a general positive mood vs no, this is wrong and a general negative mood?
Re:OpenAssistant Conversations – Democratizing Large Languag... by j45 on Sunday April 16, 2023 @12:04AM (#35587174)
This thread made me wonder if there is an AI Chatbot testing/comparison framework as of yet.