Better yet. Store the authorized public keys in LDAP or similar system. (AuthorizedKeysCommand makes this super simple these days).

That way you can rotate them easily if the private keys become compromised.

I recently built a system using this technique and synced the keys to S3, very effective.