I think this is a great question we should all think about for ourselves in advance - what does it have to do to convince you it’s actually intelligent.

Because once it does that thing without you having expressly decided that is the goal, it’s very tempting to just move the goal a liiiitle bit further away