This is precisely the realisation that is the basis of Qubes OS. The entire system is basically an admission that you cannot hope to preserve security on a system where you have users running stuff.

Just allow them to do anything, but isolate the environments such that it doesn't matter if one is compromised, because there is nothing to compromise other than the application itself.

Strongly recommended if you're willing to live without GPU support.