Parent-poster said tamper-resistance wasn't an issue in their usage case.
But are you sure it'll DFU over USB?
If so, for avoiding DFU, could you use some simple hardware to disable the data lines on the OTG port until the Pi had finished booting?
Could one use an i2c or spi based crypto chip for key storage?
An FST-01 is a somewhat better choice, but Gnuk doesn't implement U2F. If someone has enough time and knowledge I don't see why it won't be possible to add it, though.