This stuff is not fine-tuning - it's RLHF (reinforcement learning from human feedback). Basically just a lot of people marking up responses as good / bad according to the assessment criteria in the script they're given.

And yes, it is very likely that it would have seen that exactly question in RLHF. But even if not, it had seen enough to broadly "understand" what kinds of topics are sensitive and how to tiptoe around them.

RLHF is fine tuning :)

You're right, my apologies. What I meant is that specifically in ChatGPT context they have been always talking about RHLF as separate from fine-tuning on preassembled training data. As I understand, they use the latter mainly to get the desired behavior as a chatbot "eager" to answer questions and solve tasks. And then RLHF is what "puts the smiley face on top", so that it refuses to answer some questions, or gives "aligned" answers. The pronoun would be in that category.

My bet is that they use the pre-assembled training data to train the reward LLM and maybe also for fine tuning the model in addition to RLHF.

This is my best guess, afaik the research is pretty opaque right now.