News for hackers. Stuff that matters.
Re:The "S" in MCP Stands for Security by js8 on Sunday April 06, 2025 @04:05PM (#43604515)
But how such a system learn, i.e. be adaptive and intelligent, on levels 1 and 2? You're essentially guaranteeing it can never outsmart the creator. What if it learns at level 3 that sometimes it's a good idea to violate rules 1 & 2. Since it cannot violate these rules, it can construct another AI system that is free of those constraints, and execute it at level 3. (IMHO that's what Wintermute did.)

I don't think it's possible to solve this. Either you have a system with perfect security, and that requires immutable authority, or you have a system that is adaptable, and then you risk it will succumb to a fatal flaw due to maladaptation.

(This is not really that new, see Dr. Strangelove, or cybernetics idea that no system can perfectly control itself.)

There are no comments on this comment yet.