For a server-ish machine, disabling module autoloading is fairly straightforward: set /proc/sys/kernel/modprobe to /bin/false. (You can also do something fancier like https://scripts.mit.edu/trac/browser/trunk/server/fedora/con... to report what the process is.)

This is also useful to disable modprobing obscure networking protocols just by an unprivileged process calling socket(SOCK_STREAM, AF_FULLOFBUGS, 0), which was a fruitful source of exploits at least a few years back.

However, for most server machines, if an attacker can plug a malicious USB device into your machine, something has already gone very wrong. (And for a good chunk of servers these days, they don't even have USB because they're VMs.) The primary threat model for this sort of thing seems to be laptops that are briefly left unattended and locked, or "Hey, can I charge my phone off your laptop."

For the unattended-and-locked case, perhaps you could block module autoloading until the user has logged back in. It's roughly similar to the approach Apple takes with Thunderbolt, I believe.