Yes, you do need a backup way of getting in. Some websites will let you print out some one-time-use codes that you can store in a strongbox. (This is cheaper than buying two Yubikeys.)

You could also get 2 Yubikeys, but might be out of luck if the website doesn't support that.

(Also, some Android phones can act as a hardware key.)