Now, I grant that for most of us, if you are the target of any nation state level actor, not much you can do. For the most part, they can just compel the party that has your data. That said, security keys are far safer than portable data.
Maybe to secure corporate accounts against phishing, or whatever. But for my personal stuff, I need to be able to recover in many ways at any time with just a random computer and internet access without any special HW that may not be readily obtainable the moment I need it.
My personal feel is I'm ok not having my key with me when I'm "out and about." Is no different than my not having some of my ID cards with me. Such that, I'm already "locked out" of many of my financial accounts and such when I am not at home. But, I fully ack that that is a personal choice here. I just can't think of any situation I've been in, in a LONG time, where I was needing to randomly use a computer.
Edit: It also occurs to me that keeping keys to get back into your home is almost certainly similar in concern here. As somewhat crazy as I think the "keychain" nature of these tokens is, it does address a large part of this problem.
Even if you would never be tricked into giving up or losing your key, there are other people in your life who can do it either for themselves or be tricked or compelled into doing it for someone else while you're asleep or in the shower or any of 1000 opportunities every day. Would you immediately detect a good fake substitution? When would be the next important login that needed it?
Phishing is possible to avoid.