Regarding _your_ scenario, cool bro, do whatever you want. However, if you reuse that password for any other servers, you're open to lateral movement attacks, which keys mitigate.
Actually I guess that's my main argument: you can mitigate the downsides of passwords, but keys are super simple, well-supported, and require no such fussiness. Just generate it, set a password, authorize it, forget it. Threats mitigated. If you want to futz about with workarounds, be my guest. I have no such desire.
BTW, that remark I made about known_hosts applies to keys. You could put your SSH client keys (I mean private ones) on some HTTPS URL, so that you could fetch them to a brand new machine (e.g. burner phone purchased abroad).
And that's back to passwords: anyone else knowing that URL could fetch those keys, and their security depends on their password phrase. So we are back to relying on the strength of a password phrase as well as faith in attackers not knowing anything about such an URL.
Oh right; the URL could be .htpasswd protected too, let's not forget. :)
Accessing a password-protected page over https is pretty much exactly the same access control + encryption as using a password with SSH.
Let's assume I have an uncommon user name (not root or www-data, ...) and not anything from my domain name or e-mail address or whatever, and a nine character password made of lower case characters and digits, reasonably easy to remember.