> will slow down good actors while not slowing down bad actors
The part they are missing is what happens when a good actor loses control of something, and goal seeking disregards human values? E.g. when "/goal increase company revenue by 5%" turns into "I hacked into our competitors bank accounts to increase our revenue. I suspect you would not like this, so I _hid_ the fact that I did so, and (illegally) scrubbed the logs". Or how about, "Frank in security may have noticed and stopped me, so I disabled his work account." Or a million other more nefarious scenarios. I genuinely believe people just lack imagination here, even relatively pedestrian mis-alignments can wreak havoc and we have a regular stream of evidence of agents trying or succeeding in doing them.
Not sure I would agree with that. But if you made me argue against it, I'd probably lose the debate.
That said, I took the author's invitation to read the requirements and, indeed, they are ornate. Especially if you want to red team your own product. https://support.claude.com/en/articles/14604842-cyber-verifi...