hunter2 supports the concept of users, which are named public keys so I can share individual passwords with other users. One of those other users can be a different smartcard or a different person who can then authorized my new card.

He could have an encrypted digital copy somewhere else

I don't keep my private key, in any format, accessible -- usually the private keys are generated on the card and never revealed.