What do you consider to be "physical access," though? Usually that line of thinking is for when you have a motivated, intelligent attacker sitting in front of your computer. For this, "physical access" just means you plug something into your computer. It could be something you got straight from a store, and which is malicious from the manufacturer, so it seems like the only choice with that attitude is to never plug anything into your computer, or just always assume you're compromised.