There are already many legitimate apps distributed outside of Google Play for various reasons, such as weird Google policies or simply being booted out with no or spurious reason & the developer not being able to ever reach a human to fix this.
So be careful what you wish for.
I think Apple's desktop solution to unverified developers is a good way to split the difference. Deny by default but allow whitelisting. They go even further under the privacy tab and only allow certain applications permission to access accessibility features or full disk access, etc.
Maybe it's a good idea to hide the "Allow sideloaded apps" under the developer menu in Android or something, or generally to display a scarier message.
This and it's ability to survive factory reset may indicate, that xhelper can gain complete control over device (probably via improperly built firmware or unpatched root exploits). No amount of sandbox enhancements can stop this kind of priviledge escalation.