So everyone's safe or no one is safe?

That could be the case.

I’m not arguing that you shouldn’t allow side loading.

But let’s say this is what developers start doing. We compile our own code and side load. Great for us. How many people can each individual support? Immediate family and close friends? Are most people on their own? Now say your immediate contacts are compromised. That exposes some of your details as well would it not? At least you’d be more vulnerable to social engineering.

So maybe you have to work towards a system where everyone is safe.

theres no other way in general, those who have no clue about things will be doomed to not know how to conduct themselves in a secure manner

I think a good middle-ground would be having the default store have the functionality for verifiable builds like F-Droid. Add a banner/badge that the app you're looking at is open source, a red flag if it's closed source, a link to the release page in their code repository for that specific release, so that techie-people can verify the hashsum themselves and look at the code.

Of course, there's still plenty of possible supply-chain attacks and the like. Closed source app store, git repositories taken over or sneaking in malicious commits, binary blobs required to run your device, and so on. But we should encourage any progression in security, and turn good security practices into common sense.

It's possible that Apple's new Xcode Cloud service could support this at some point, providing a continuous pipeline from source to release. Though I doubt it is a high priority for them.