> Or if someone gets a hold of my phone and now I have a brand new app with all permissions installed.

System permissions aren't tied to the App Store as far as I know (same on Android). You still need to give apps permission to access your camera, mic, location, etc.

While allowing sideloading would bypass Apple's review process (which can be bad), if someone has that type of access to your phone, there's nothing stopping them from installing tracking/spyware apps from the App Store right now.