Yeah, I've always said that AI isn't AI if it needs safeguards that are so extreme it cannot figure out if something is genuinely bad or not.
Think of like calling someone's workplace asking when their next shift is. There are a lot of good reasons someone might want to know this that the worker would be ok with, delivering a gift or whatever. Or it could be someone intending to harm them. We've mostly figured it's not worth having the person on the phone try to figure out in the moment if this is genuinely bad or not, so they follow the "safeguard" of just not giving the information.
The problem here isn't really that safeguards exist on language models. They pretty much have to to some extent for most uses. The problem is more in who controls the policy.
If they make questionable decisions simply have your Butler send for your pet congresscritter and let them all hash it out with your staff over cucumber sandwiches. Failing that you can just take your private space-yacht to mars with Elon.
Failing to recognize this distinction is the danger: Claude seems very good and wins people over easily with its earnest principles. But power belongs to Anthropic.
They even acknowledge that in the preamble:
> Training models is a difficult task, and Claude’s behavior might not always reflect the constitution’s ideals.
closed models perpetrated the attack
That's the more important part. They are clearly wanting to have super special tier for "trusted" clients (realistically, just ones paying a lot) that have access to those capabilities and rest of the plebs can't even use it to secure the app they own.
It's a protection racket, pay us, or the security firm that pays us a lot of money to get access to the fixes for the AI-found attacks early
More importantly, it showcases how little teeth the Claude constitution has. Claude will reassure users that its constitution means it prioritizes user autonomy and privacy, but will quickly admit how meaningless that is when "safeguards" override Claude.
That is to say, the constitutional approach to Claude seems weak and even misleading if said constitution puts no constraint on Anthropic.