In the case of ssh keys, you usually should use a different key per device/home directory and let your server accept all the keys.