You're probably overthinking it. Use SSH keys to log in, and if you want to lock things down further than "admin users are simply in sudoers", then write a more complicated sudoers that permits only some commands for some users.

In practice, an attacker who gets a shell on one of your servers has game-over access to the data center the server's in.