What's particularly nice is if you can plug in a classifier for things like esthetics based on human ratings, along the lines of 'learning from human preferences' https://arxiv.org/abs/1909.08593 but better - why spend the enormous effort running PPO to brute force the classifier to obtain desired text or image output, when you can just backprop through it and let the classifier itself tell you how exactly to improve the inputs?

(I was writing up this idea as a proposal for improving preference learning at https://www.gwern.net/GPT-2-preference-learning#optimization... based on my experiences idly waiting for PPO to run and being annoyed by the divergences, and then Uber just... blogs it out. Like a madman.)