Its turtles all the way down. You need an anchor of trust. A trust root. This is the public PKI system trusted root store.

Even if you obtain the self-signed cert out of band (and explicitly trust it), how do you authenticate that channel?

Self-signed certs are not scalable or particularly useful for internet users. Please don't recommend this.

Like a public certificate authority?

Maybe we could design a protocol for securing the socket layer, maybe even automate the key exchange so that it's basically transparent to the user, and then why not do the same thing for the people that need certs, let them ask for it whenever they want and provide them a nice tool to automatically renew it. /s