I think the biggest problem is whether Claude could be tricked into doing so. I could see how mass surveillance could be repacked as "summarize my conversations", or autonomous killbots could be playing a video game.