Well, the simple objection would be that the more remote network services you include in admin login management,the more likely you're going to have a problem when you least can afford it.

SSH configs in distros are moving towards not even doing rDNS lookups (which generally just delay logins for the timeout period when there's a problem).

That doesn't mean some hybrid solution doesn't have merit though.