You're describing an overly optimistic scenario. "Why not just harden all your machines?" Because it's more complicated than that. Yes, you can automate basic hardening (configuration of access controls, limited use of SSH keys), but how do you automate hardening of RPC between one server and another, or hardening of those 3 internal administration web apps which may or may not have some local file disclosure and crypto flaws?
Defense-in-depth is still the only sensible way to handle security. You do what you can on the host side, but bastion hosts, jump hosts, VPNs, etc. provide an additional layer to make lateral movement much more difficult for an attacker.
Better yet. Store the authorized public keys in LDAP or similar system. (AuthorizedKeysCommand makes this super simple these days).
That way you can rotate them easily if the private keys become compromised.
I recently built a system using this technique and synced the keys to S3, very effective.
Then you avoid having a single point of failure. If and when a single SSH key is compromised, there are a limited number of hosts that key had access to - and unless that's the log server, you have secured logs so you can figure out what happened. Different keys can have different capabilities, rather than a one-size-fits-all policy on the bastion. It makes it harder for an attacker to know where to get the boss's keys. And you can expand your system geographically without the performance compromises of a VPN; your servers already make encrypted, secured calls to each other (over SSH, SSL or a very small number of vetted protocols), expose only those ports that are actively used. Your attacker could steal your entire system documentation and it wouldn't make any difference, because the system is truly secured, not relying on obscurity.