That assumes that you have perfect key management. Looking at current cases about phone & other device searches suggest that most people would be compromised.

This also forces other issues: e.g. offline wallets are a standard answer to security concerns but those are more vulnerable to physical compromise. This scenario needs a lot more discussion of threat models.