I agree it's probably simple to lock down the possible USB devices (ideally, don't even compile-in support for them in the kernel), but I am wondering if anyone can confirm it's been done, and I would still worry about exploits in the USB stack or hardware. It seems like this is the biggest chink in the armor, so I'd go for hand-keying from paper (a redundant encoding can probably make this not excruciating for the humans involved), another optical disc (CD-R), a floppy disk, a camera for scanning something like QR codes, a simple wired serial link, serial over infrared. I suppose these all have some tradeoffs, but at least some of them don't involve plugging in another electronic device that uses a complex general-purpose protocol just to transfer what probably amounts to a few dozen kilobytes.

I can see it now: A hacker with tiny QR code imprinted cuff links or rings. They brush their hand close to the camera, seemingly by accident, and BOOM! Buffer overflow in the QR decoder :)