What's surprising to me is that this would surely be in OpenAI's interests, too -- free RLHF!

Of course there would be the risk of adversaries giving bogus feedback, but my gut says it's relatively straightforward to filter out most of this muck.