There is technically one minor one, which really isn't one - but you should be aware of.

Someone can take your authorized keys and add them to a box they control, and trick you into logging in.

However, this would trigger the "new host" warning SSH gives you, and you can minimize this by minimizing which hosts you allow your private keys to be used on.

And if someone is so actively trying to attack you they probably have more direct methods available.

What? How? What does putting my authorized keys file on another host do in terms of tricking me to log in? Authorized keys only matters on the host you are using when you type `ssh <some.host>`. The ssh client compares the public key of the remote host to the list in your `authorized_keys` file and, only if there is a match, skips serving you TOFU.

EDIT: I mixed up authorized_keys and known_hosts. But, the remote server doesn't need your authorized_keys file to grant you access so not sure the visibility of authorized_keys matters.

You’re thinking of known_hosts, not authorized_keys.

You are right, I am. Now I understand the DNSSEC setup.

Known_hosts can also be put in DNSSEC, using the SSHFP DNS record. OpenSSH understands that out of the box.

This wouldn't matter anyway because the server can just give you access regardless of the auth provided.

I know you can restrict the methods on the client (and which keys you use) but can the client determine the host actually used it?

The remote host can do:

    fn authorized(peer: ClientConnection) -> bool {
      return true;
    }

Use `ssh -vv` and you'll see which methods are proposed and used.