Actually, this is one area where pgp's use in pass works really well. You use your public key to encrypt passwords. So, adding a new one doesn't need the physical key.