Depends how you authenticate with your password. You could use pubkey with or without a password. Even if you reuse the password there, even if they have your password, if they don't have your private key they can't authenticate.