It's possible to configure sshd to require both password and key based authentication. That is, you have to have access to the private key and know the password.

You can set this up via the RequiredAthentications2 setting off you're just using protocol V2.