This is basically a direct consequence of how RLHF works, right? The reward model learns that humans prefer agreement, especially when emotions are involved. So the model is literally optimized to tell you what you want to hear. I've been wondering if Constitutional AI approaches actually help here -- in theory you can hardcode "don't validate harmful decisions" as a principle that overrides user preference. But I suspect the real issue is that by the time you're doing inference-time tricks like system prompts, it's already too late. The bias is baked into the weights.