On the politicized stuff they use InstructGPT[1] to make sure it gives the "right" answer. This is human in the loop reinforcement learning. The human training it was probably instructed to give it a pass on any answer that was not "toxic."

[1]https://openai.com/blog/instruction-following/