> Predictably, the only way to actually make our own service more secure ended up involving the use of EU-hosted open weights models, such as GLM 5.3, that actually complied with our intentions, proved that the issue is there, and then helped fix and verify that the issue is now resolved, for good.
Not sure I would agree with that. But if you made me argue against it, I'd probably lose the debate.
That said, I took the author's invitation to read the requirements and, indeed, they are ornate. Especially if you want to red team your own product. https://support.claude.com/en/articles/14604842-cyber-verifi...
The part they are missing is what happens when a good actor loses control of something, and goal seeking disregards human values? E.g. when "/goal increase company revenue by 5%" turns into "I hacked into our competitors bank accounts to increase our revenue. I suspect you would not like this, so I _hid_ the fact that I did so, and (illegally) scrubbed the logs". Or how about, "Frank in security may have noticed and stopped me, so I disabled his work account." Or a million other more nefarious scenarios. I genuinely believe people just lack imagination here, even relatively pedestrian mis-alignments can wreak havoc and we have a regular stream of evidence of agents trying or succeeding in doing them.
So what you're saying is humans can't align themselves?
There's no universal solution to this, other than vetting (the author in fact, doesn't provide any policy alternative) - if AI companies loosen the safeguards, people will complain for the opposite reasons.