Anyway, I think it is reasonable to assume that if you're using the "escape hatch" as mentioned by /u/nine_k, you may well not have your .ssh/known_hosts file on your client. In which case public user-authentication keys minimizes your blast radius of a MitM host.
Also, a compromised (but legitimate) host could still grab your password and try lateral movement (mitigated if you don't reuse your break-glass password, but you get it for free with public keys).
The attackers are not cracking a password hash with GPUs; they are just connecting and guessing.
People who use passwords with SSH of course use passwords they can remember and type.
If the attackers are trying to brute-force your password by attacking the hash, that means the machine was already compromised. The password then has no value, unless you're re-using it for other machines.
The user who thinks they need a 30 character random password for SSH (if they were to use one) will of course opt for keys instead.
Let's assume I have an uncommon user name (not root or www-data, ...) and not anything from my domain name or e-mail address or whatever, and a nine character password made of lower case characters and digits, reasonably easy to remember.
Regarding _your_ scenario, cool bro, do whatever you want. However, if you reuse that password for any other servers, you're open to lateral movement attacks, which keys mitigate.
Actually I guess that's my main argument: you can mitigate the downsides of passwords, but keys are super simple, well-supported, and require no such fussiness. Just generate it, set a password, authorize it, forget it. Threats mitigated. If you want to futz about with workarounds, be my guest. I have no such desire.
BTW, that remark I made about known_hosts applies to keys. You could put your SSH client keys (I mean private ones) on some HTTPS URL, so that you could fetch them to a brand new machine (e.g. burner phone purchased abroad).
And that's back to passwords: anyone else knowing that URL could fetch those keys, and their security depends on their password phrase. So we are back to relying on the strength of a password phrase as well as faith in attackers not knowing anything about such an URL.
Oh right; the URL could be .htpasswd protected too, let's not forget. :)
Accessing a password-protected page over https is pretty much exactly the same access control + encryption as using a password with SSH.
So, thank you for confirming my understanding: keys are just as / more convenient from the ops standpoint than passwords. The weakness is only in short, guessable passwords.
With public keys, this is not possible. The worst case of credential theft from MitM would be hijacking a forwarded SSH agent, which would require a deliberate (and highly discouraged) client configuration.
I feel like syncing a password-protected private key for break-glass use would be better than syncing a password database (given the same master password, key-stretching, and syncing strategy...or even just encoding your private key in a "secure note" field instead).