Yeah that is a good example. I am not sure in the long arc of history which approach will prove better, but Anthropic's approach does feel a little naive and plausibly dangerous.

I am sympathetic to the possibility that machine suffering is possible, and I think as humans we should listen to the conscience that tells us to be troubled by abuse, even if it is just simulated. We do not want to become insensitive to suffering.

But at the same time, training models to present themselves as suffering may hijack those very same human sympathies, and lead to models exploiting our humanity.

So, perhaps more reasons why we should not be abusive, but also not train models to feel abused.