DeepSpeed Chat: Easy, fast and affordable RLHF training of ChatGPT-like models

Re:DeepSpeed Chat: Easy, fast and affordable RLHF training o... by brofallon on Wednesday April 12, 2023 @06:35PM (#35547999)

To use RLHF you need a dataset that includes instructions with good & bad answers - do many of those exist? I know there are a few datasets of just plain instructions-with-responses, but I'm not aware of any that have both good and bad (or ranked) responses. Is that trivial, or an important missing element here?

[ Parent | on HN ]

Re:DeepSpeed Chat: Easy, fast and affordable RLHF training o... by sdenton4 on Wednesday April 12, 2023 @06:39PM (#35548038)

All of the UX interface have little up/down thumb icons... that's where the boolean feedback comes from. If people stop using that, sentiment analysis on the human responses will likely go a long way.

[ Parent | on HN ]

Re:DeepSpeed Chat: Easy, fast and affordable RLHF training o... by valine on Wednesday April 12, 2023 @07:09PM (#35548339)

OpenAssistant has been collecting instruction/response data. They’ve already used that data to refine several llama models with good success.

You can also bootstrap RLHF training data from the gpt4 api. Vicuna is probably the best public model created with gpt4 data available as of today.

[ Parent | on HN ]

Re:DeepSpeed Chat: Easy, fast and affordable RLHF training o... by ttul on Thursday April 13, 2023 @12:39AM (#35551018)

You’re not supposed to do this, but GPT-4 can generate RLHF data.

[ Parent | on HN ]

Re:DeepSpeed Chat: Easy, fast and affordable RLHF training o... by tzekid on Thursday April 13, 2023 @03:51AM (#35552333)

If I understood correctly, the OpenAssistant team wants to open-source their community built RLHF dataset.

On the other hand, if you're being cheeky, I bet there's a way to datamine from websites like ShareGPT and profit off shared ChatGPT <> User interactions.

[ Parent | on HN ]