That's in generally available distro a huge PITA.

You can do blacklists easy enough if you want to, just add few lines of text into /etc.

I'd also like option for whitelisting, like whitelisting every single NIC driver is harmless enough coz they just won't be loaded, but anything that can be loaded by non-root userspace action should have option to be only loaded if it is on whitelist.

Tho all that is easily doable by just changing userspace AFAIK