I'm sure a udev rule would be sufficient to whitelist only the trusted hardware. However, nothing can stop a malicious one to spoof serial numbers or other characteristics of a trusted hardware.