The "something you have" shouldn't be easily replicated. If someone could capture your SSH key passphrase and copy the ssh private key then it's not an actual 2FA. a device like a Yubikey won't let you extract the stored keys, so the only way to gain access is to steal the physical key and that makes the attack much harder to do stealthily.