Yes but these were steps were not used in R1-zero where its reasoning capabilities were trained.
Re:RLHF Book
by littlestymaar on Sunday February 02, 2025 @05:28AM
(#42907748)
And as a result R1-zero is way too crude to be used directly, which is a good indication that it remains relevant.
In addition, the "reasoning-oriented reinforcement learning" step (section 2.3.2) used an approach that is almost identical to RLHF in theory and implementation. The main difference is that they used a rule-based reward system, rather than a model trained on human preference data.
If you want to train a model like DeepSeek-R1, you'll need to know the fundamentals of reinforcement learning on language models, including RLHF.
[1] https://arxiv.org/pdf/2501.12948