Again, Internet security is mostly based on trust and there's no way around that: when you enter a URL on a browser, you need to trust that the domain belongs to who you think it belongs, and the only way you have to do that is to trust its TLS certificate... which you do by explicitly trusting your browser to only recognize reliable TLS certificates signed by a legit root certificate authority... unless you decide to whitelist TLS certificates yourself by obtaining the domain's root certificates out-of-band, something nobody seems to think necessary to do, even the most security-conscious government agencies probably go to that kidn of trouble.

Basing package ownership on the same kind of mechanism is the best you can ever do... if you consider that as not being a good enough solution, then you should never use the internet with consumer-grade browsers at all, let alone use package managers in any form.