I agree with this assessment, and it's what I do. There is still the single point of failure of losing your domain due to a hostile registrar or mismanagement e.g. allowing registration to lapse.

Ideally there would be some a decentralized permanent domain registry keyed on certs (I know these exist but have not been adopted), or at least a fallback domain you could configure somehow in case you lost control of your main domain.