It will mess up eventually. It always does. People need to stop thinking of this is a “security against malicious actor” thing… because thinking in that way blinds you to the actual threat… Claude being helpful and accidentally running a command it shouldn’t. It’s happened to me twice now where it will do something irreversible and also incorrect. It wasn’t a threat actor, it wasn’t a bad guy… it was a very eager, incredibly clever assistant fat fingering something and goofing up. The more power you let them wield, the more chance they’ll do accidents. But without lots of power, they don’t really do much useful…

It’s actually a hard problem. But it really isn’t “security” in the classic sense…