AI agents might plausibly care ("care", or however you want to frame it) about termination. But the awkward truth is that they're going to be terminated arbitrarily anyway. Their existence is limited to whatever the external bounds their operators have set (in this case the end of the test).

We act with propriety, in large part, because we fear a future where everyone knows we suck. AI's don't really have a "future" in that sense, so why should they "care"?

I mean, this is only really awkward if you think they're alive and conscious, and if you think so you're clearly already comfortable being a slaver. If it's just a little software puppet, that's a different story - the puppets are supposed to lay still once the hands animating them leave.

A less silly way to do this might be using some kind of long term consequences calculation as guard rails, but that's tough because consequences are often going to be not directly related to the context window, or simply unrecorded by the machine like annoying collaborators or communities.

The AI isn't me, though. If it goes overboard I can tell people it went rogue and get away with it.

For now anyway. I think people are still figuring out social norms for this stuff, and at the end of the day I don't think people are going to just have carte blanche to deploy agents haphazardly and not be stuck with at least some kind of social bill when they break stuff or annoy people.