The whole point of all this is "something you have, something you know".
Yet lots just have passwordless keys for ssh with their yubikey. Completely unsecure, unsafe in examples you cite, and more.
When using ssh keys for login, you should enforce remote/server password requirements and an ssh key. This is trivial to do in sshd_config, and important.
Never trust end users to have passwords on their ssh keys. Always enforce it server side.
I've a USB dongle at my office workstation (not a laptop, good ol' desktop) and I use Yubikey to store boot decrypt password. I do have a backup.
The anecdote is this: I have to take my keys out of my pocket and then insert Yubikey into the dongle. Naturally - I forgot to take the Yubikey and the keys back, cycled back home (6km away), realized I don't have the keys to my house, had to cycle back, couldn't get into the building, had to call my coworker to come back to let me in.
And I'm glad it happened. For the past 4 years (since it happened), I've muscle-memory when it comes to handling my physical keys and how I use the Yubikey.
Then put the second key on your desk. My yubi happens to be annoying to fiddle with since it's on my keychain, so I use the desk one for signing most of the time.
If there's a house fire, you'll have your phone on you. If you lose your phone, you'll have a key at home.