Its a good thing to think about, but I don't see it as a huge problem. I had a Yubikey Nano plugged into my laptop almost constantly and I do trigger the OTP sometimes, but using that as an attack vector is pretty hard, specially for all the sticks that are not always plugged in.