News for hackers. Stuff that matters.
Re:OpenAI Reinforcement Fine-Tuning Research Program by radarsat1 on Saturday December 07, 2024 @08:11AM (#42349587)
If I'd like to learn more about DPO and RLHF, I've been looking for toy problems/datasets to use but coming up a bit empty handed. Is there a convenient way to experiment with these methods through toy problems and simulation that can be done on a single GPU? The need for massive data and parameter counts to do anything interesting makes learning about these methods a little daunting.
Re:OpenAI Reinforcement Fine-Tuning Research Program by roborovskis on Saturday December 07, 2024 @10:08AM (#42350247)
https://stable-baselines3.readthedocs.io/en/master/ is a great resource for hacking on implementations for RL - many good RL courses out there but https://www.youtube.com/playlist?list=PLwRJQ4m4UJjNymuBM9Rdm... is my personal favorite.

For LLMs / RLHF it's a little more difficult but https://github.com/huggingface/alignment-handbook and the Zephyr project is a good collection of model / dataset / script that is easy to follow.

I would suggest studying the basics of RL first before diving into LLM RLHF, which is much harder to learn on a single GPU.

Re:OpenAI Reinforcement Fine-Tuning Research Program by radarsat1 on Sunday December 08, 2024 @04:18AM (#42356029)
Hi, the Zephyr link may be what I'm looking for. yeah I'm quite familiar with RL already so it was specifically RLHF that I was asking about, I'll check out that resource, thanks!