> has almost no impact on the average user security

I don't think that's true.

Google publishes malware stats on the on-play and off-play ecosystems and the malware rates differ. Famous examples like malicious versions of FortNite make news headlines. Play policies also enforce various security-relevant action like updating to near-recent targetSdkVersions and fixing a variety of automatically detectable security vulns. No such enforcement exists in the broader off-play ecosystem so it will be easier to install an app with trivial security flaws.

Whether this is a justification for banning sideloading completely? That's another question.