I wonder if you could make a setup where you have actual human volunteers in a vr environment, have the agent instruct the human on what action to take, and then reward the agent on the correspondence between the instruction and the human’s behavior. Maybe there would be too many degrees of freedom in the actions humans could take for this to be useful. Also, the setup has clearly dystopian elements.