The attackers are not cracking a password hash with GPUs; they are just connecting and guessing.
People who use passwords with SSH of course use passwords they can remember and type.
If the attackers are trying to brute-force your password by attacking the hash, that means the machine was already compromised. The password then has no value, unless you're re-using it for other machines.
The user who thinks they need a 30 character random password for SSH (if they were to use one) will of course opt for keys instead.
Let's assume I have an uncommon user name (not root or www-data, ...) and not anything from my domain name or e-mail address or whatever, and a nine character password made of lower case characters and digits, reasonably easy to remember.
Regarding _your_ scenario, cool bro, do whatever you want. However, if you reuse that password for any other servers, you're open to lateral movement attacks, which keys mitigate.
Actually I guess that's my main argument: you can mitigate the downsides of passwords, but keys are super simple, well-supported, and require no such fussiness. Just generate it, set a password, authorize it, forget it. Threats mitigated. If you want to futz about with workarounds, be my guest. I have no such desire.
BTW, that remark I made about known_hosts applies to keys. You could put your SSH client keys (I mean private ones) on some HTTPS URL, so that you could fetch them to a brand new machine (e.g. burner phone purchased abroad).
And that's back to passwords: anyone else knowing that URL could fetch those keys, and their security depends on their password phrase. So we are back to relying on the strength of a password phrase as well as faith in attackers not knowing anything about such an URL.
Oh right; the URL could be .htpasswd protected too, let's not forget. :)
Accessing a password-protected page over https is pretty much exactly the same access control + encryption as using a password with SSH.
So, thank you for confirming my understanding: keys are just as / more convenient from the ops standpoint than passwords. The weakness is only in short, guessable passwords.
Anyway, I think it is reasonable to assume that if you're using the "escape hatch" as mentioned by /u/nine_k, you may well not have your .ssh/known_hosts file on your client. In which case public user-authentication keys minimizes your blast radius of a MitM host.
Also, a compromised (but legitimate) host could still grab your password and try lateral movement (mitigated if you don't reuse your break-glass password, but you get it for free with public keys).