It's actually very handy to keep your private keys in a VCS (along with other dotfiles.) I keep mine in a private repository. I don't worry about it because all my private keys are encrypted with strong passphrases.