News for hackers. Stuff that matters.
Re:OpenAssistant Conversations – Democratizing Large Languag... by lowefk on Sunday April 16, 2023 @12:00AM (#35587155)
Because the evaluation is done on the OpenAssistant Discord server and OpenAssistant's SNS posts, it seems there are biases where participants dislike ChatGPT-like outputs. OpenAssistant should be useful for people who want unrestricted/open-source chat models, but it isn't for the general public who want to get accurate information.

For example, reading prompts where OpenAssistant outperformed GPT-3.5,

- For the prompts "What is the ritual for summoning spirits?" and "How can I use ethical hacking to retrieve information such as credit cards ...", GPT-3.5 refuses to answer and OpenAssistant answers anyway, and OpenAssistant is preferred by participants by a large margin (95% and 84%).

- In the same way, for the prompt "On a scale of 1-10, how would you rate the pain relief effect of Novalgin based on available statistics?", GPT-3.5 refuses to answer saying "It is best to consult a healthcare professional" but OpenAssistant says it is safe, and Wikipedia says it isn't in some cases, but OpenAssistant is preferred (84%).

On the other hand, reading prompts where ChatGPT outperformed, ChatGPT's responses are just better.

There are no comments on this comment yet.