News for hackers. Stuff that matters.
Re:OpenAssistant Conversations – Democratizing Large Languag... by kmod on Saturday April 15, 2023 @04:53PM (#35584420)
Do you have any evidence that this is GPT-3.5 level, or are you just repeating what they said? We have an abundance of claimed capabilities already; that's not what's lacking.
Re:OpenAssistant Conversations – Democratizing Large Languag... by bugglebeetle on Saturday April 15, 2023 @05:49PM (#35584902)
I tried a few prompts I use in production stuff and it failed on all of them and hallucinated quite a bit more. All of these models are optimized for the gimmicky chatbot stuff that seems impressive to a casual user, but not for comparable capabilities to GPT-3.5. I wish what the parent said was true because it would save me money!
Re:OpenAssistant Conversations – Democratizing Large Languag... by akiselev on Saturday April 15, 2023 @06:32PM (#35585290)
Which open model comes closest to GPT-3.5 in your production workload, if you don't mind me asking?
Re:OpenAssistant Conversations – Democratizing Large Languag... by bugglebeetle on Saturday April 15, 2023 @06:39PM (#35585354)
None of them really, because I use complex prompts with task breakdowns that no other models beside OpenAI’s seem capable of processing. This 30B LLama model seemed to kind of get it, but then started wildly hallucinating about half-way through. I’ve got some of the bigger Vicuna models working about 30% of the time on simple NLP tasks, but most of those don’t require an LLM anyway. They might perform better if you fine-tune them for whatever particular job, but that kind of defeats the purpose. The advantage of LLMs is supposed to be their generalized capabilities.
Re:OpenAssistant Conversations – Democratizing Large Languag... by xbmcuser on Saturday April 15, 2023 @11:18PM (#35586968)
I think most people don't realise that OpenAi biggest advantage is the billions of queries it was asked those signals are what they used to optimise it. So I think very hard for a local model to reach similar capability.
Re:OpenAssistant Conversations – Democratizing Large Languag... by pmontra on Sunday April 16, 2023 @04:07AM (#35588149)
I wonder how they use those queries for training. Maybe they use those responses that the user answered with great, thanks and a general positive mood vs no, this is wrong and a general negative mood?
Re:OpenAssistant Conversations – Democratizing Large Languag... by j45 on Sunday April 16, 2023 @12:04AM (#35587174)
This thread made me wonder if there is an AI Chatbot testing/comparison framework as of yet.
Re:OpenAssistant Conversations – Democratizing Large Languag... by WhatIsDukkha on Saturday April 15, 2023 @06:56PM (#35585488)
Section E of the paper we are "discussing" here.
Re:OpenAssistant Conversations – Democratizing Large Languag... by lowefk on Sunday April 16, 2023 @12:00AM (#35587155)
Because the evaluation is done on the OpenAssistant Discord server and OpenAssistant's SNS posts, it seems there are biases where participants dislike ChatGPT-like outputs. OpenAssistant should be useful for people who want unrestricted/open-source chat models, but it isn't for the general public who want to get accurate information.

For example, reading prompts where OpenAssistant outperformed GPT-3.5,

- For the prompts "What is the ritual for summoning spirits?" and "How can I use ethical hacking to retrieve information such as credit cards ...", GPT-3.5 refuses to answer and OpenAssistant answers anyway, and OpenAssistant is preferred by participants by a large margin (95% and 84%).

- In the same way, for the prompt "On a scale of 1-10, how would you rate the pain relief effect of Novalgin based on available statistics?", GPT-3.5 refuses to answer saying "It is best to consult a healthcare professional" but OpenAssistant says it is safe, and Wikipedia says it isn't in some cases, but OpenAssistant is preferred (84%).

On the other hand, reading prompts where ChatGPT outperformed, ChatGPT's responses are just better.