I have everything in ~/dotfiles which is a git repository here: https://github.com/dmarcoux/dotfiles. Basically, I use GNU Stow to symlink what I need. I have master which contains common config and one branch per computer. Everything is explained in the README, in more details if you want do know more. It's not perfect yet, I still have some small irritants, but I'm quite happy with this setup.