There are downsides for Apple to the rate of their 30% cut. That is why they are doing their best to keep sideloading off their devices. Not to protect users - as you already stated there are many other layers of protection in place for that - but to protect their revenue.
Sandbox escapes are incredibly common, Apple still controls eliminating your business from iphones if you attempt to use an exploit to pull user data when you distribute through the App Store.
If you can do distribution via your own site, it becomes a whack-a-mole game with Apple - where data exfiltration exploits are found, Apple fixes them in a new update, then we have to wait for people to update.
And until most users update, the at-fault company is busy siphoning data from any user who can't update (e.g. no wifi / limited data plan) or any user who is slow to approve the update dialog. Eventually the company's app will grow to contain an exploit for the last dozen iOS versions that conditionally execute based on the iOS version/feature detection/probing for exploit availability.
If iOS' sandbox is that insecure, then they have a problem on their hands.
This is why a malware app on an iPhone can only do limited damage. It can't access all files, it can't encrypt the storage, it can't launch a DDoS in the background because the OS doesn't let it run, etc. iOS or Android are not Windows.
If you want to know how it works in practice, look at Android. It has supported sideloading for a long, long time. People do install a lot of crap, but that's from the app store. My parents are really bad with tech and never sideloaded anything.
And there are other layers of security too. If you go to a store, buy a Samsung, a Google Pixel, a OnePlus, Nokia, etc, they all come with Google Services, which includes Google Play Protect... essentially an anti-virus that looks at your apps and flags anything that is known to be malicious.
Are there any downsides? Yes. Are they as bad as some say? The Android example tells us that it doesn't have to be that bad.