In the case you are mugged, that USB key is going with the mugger. (see also police raids or airport "inspections" e.g., entering or leaving bad states) I prefer memorized keys, with, ideally, a duress passphrase for this reason.

But, hypothetical scenarios without reasonable threat modelling is a losing man's game. You can never one-up a mythical adversary. Protecting against the easy 80% of threats takes only 20% effort.