However, that could be bypassed if the USB key acted like a USB hub with a keyboard attached.
If you do get a keyboard working, it looks like you can just log in with raspberry / raspberry and then full privileged access since sudo from the raspberry user, and /dev/kmem from root is enabled. That would allow arbitrary content to be copied on to the user supplied USB, and depending on the device, possibly firmware reprogramming.
I'd say just don't run a graphical environment or, if you do, lock it. With a normal password, good luck to the USB thumb drive that pretends to be a keyboard.
As for booting, well, don't have it in there at boot time.
As for malware in the files itself -- I don't think that's the point of this project. It says "if you don't trust the transport medium (i.e. USB)", it's not about the contents.