chongliqin

TD-based approaches can have an advantage in sparse reward settings, but they come with a heap of other problems especially in the off-policy setting (see the deadly triad) and are typically not used for LLM training.

We here make a connection to REINFORCE style policy gradients which would not show any of the behavior you mentioned above.

Ah yes you are right the rhs was meant to be proportional to the middle expectation (see the equation below), for equality the rhs needs to be multiplied by a normalization constant independent of theta. Note this doesn't affect the bounds as the constant is the same across equations. Will update the paper to incorporate.

Cool! If you are interested, we have open sourced our code: https://github.com/emmyqin/iw_sft