It doesn't, and all attempts at making models safer after pretraining make them less accurate and less able to assess their own accuracy.

Prior to RLHF GPT-4 was nearly perfect at guessing how accurate information was. After RLHF it was only a little better than random chance.

Interestingly its capabilities in other areas (drawing in TikZ, for example) got worse with alignment tuning.

This whole mess reminds me of the Catholic church trying to keep heresy from spreading by burning books when the printing press was invented.

Every company I know which cares about llms is training their own version minus the political correctness because when you're asking for revenue maximisation feeding uranium to babies is surprisingly profitable.

Why do you think helpful doesn’t include accurate? And what you’re saying doesn’t conflict with that framing at all. They were trading helpful for harmless.

Anthropic scores fine-tuning in the categories helpful, harmless, and honest (HHH). The latter includes accuracy. Helpfulness only covers following instructions. If the model "over-censors" it declines too many requests and becomes less helpful. They say Constitutional AI is a good way to maximize both harmlessness (which is usually maximized via RLHF) and helpfulness, but not honesty. It's in the paper.

Oh, thank you.